Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchNo close match found

End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic Workflows

Build a benchmark suite pairing raw molecular, sequence, and spectral inputs with multi-step software tools to evaluate whether multimodal agents actually use scientific perception during discovery.

Time

a couple of months

Compute level

Multiple GPUs

Resources

Multi-GPU inference cluster capable of serving the 320B MoE model (e.g., 8x H100/H800 GPUs), CPU container infrastructure, and public scientific datasets/software.

What the paper used

The paper reports a 320-billion-parameter MoE model, ToolSpace data covering 77,760 tool-call pattern nodes across 2,042 tools and 804 environments, and ML Coding candidate inventories of 2,960 papers and 5,636 dataset sources, but does not state the GPU hardware or training compute budget used for pretraining or post-training (Appendix B only specifies 1-4 CPUs and 8-16 GiB memory per task container during benchmark evaluation).

Prerequisites

Multimodal LLM evaluation, bioinformatics/cheminformatics toolkits, containerized agent harnesses (Harbor/XTuner), and benchmark design.

Scope

new benchmark suite and ablation pipeline

The Gap

In Sections 2.2 and 3.2, the paper trains modality-specific projectors (Sage Sci-Perception) for molecular graphs, biological sequences, and spectra using SFT and GRPO on standalone scientific reasoning and auxiliary evidence-querying tasks. However, the formal 33-benchmark evaluation in Section 4 (Table 2) tests only text, code, and GUI/visual tasks (e.g., OfficeQA Pro, MLE-bench Lite, OSWorld-Verified), while Section 5 presents only qualitative case studies (CT reconstruction, 3D modeling, Fiji/EEGLAB/QGIS, and HTML physics demos) that do not invoke the scientific encoders.

“A next step is to test these capabilities together in complete scientific workflows, checking whether the model selects relevant evidence, adapts its actions to feedback, and produces scientifically correct results.”
From the paper, Section 6, page 23. Matched against the PDF text.

Limitation

Because scientific perception adaptation and agentic tool use are evaluated separately, there is no quantitative evidence showing whether projected scientific embeddings remain usable, accurate, and grounded when combined with long-horizon tool calling, software manipulation, and visual feedback in real scientific workflows.

Proposed Solution

Construct a containerized benchmark suite in the Harbor format used by the authors that supplies both raw structured inputs (molecular graphs, protein/DNA/RNA sequences, spectra) and domain software tools (e.g., RDKit, Biopython, 3D Slicer, EEGLAB, Fiji) in the same episode. Design tasks requiring the agent to inspect native scientific embeddings to formulate hypotheses, execute software transformations or simulations, compare experimental outputs against source measurements, and pass deterministic artifact and scientific-correctness checks, evaluating both full-interface and text-only ablated baselines.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

If successful, this benchmark and evaluation protocol would verify whether native scientific encoders genuinely improve multi-step scientific execution over text-only or image-only representations and expose cross-modal interference during agentic rollouts.

How to Get Started

How to test it: Hypothesis: Native scientific embeddings outperform text/SMILES-only or rendered-plot baselines when agents must localize subtle structural features and invoke domain software tools. Baseline: Sage S2.1 Flash with Sage Sci-Perception disabled (text/image inputs only) on a 100-task Harbor scientific workflow benchmark measuring task pass rate and scientific-correctness F1; stop if native scientific embeddings fail to beat the ablated baseline by at least 5 percentage points across modalities.

  1. Inspect the released InternLM/Sage-S2.1-Flash and InternLM/Sage-SciPerception repositories to confirm how scientific tokens are formatted alongside tool-use prompts.
  2. Curate 50-100 multi-step tasks from public cheminformatics, protein-engineering, and spectroscopy datasets requiring both raw scientific inputs and executable Python/CLI tools packaged in Harbor format.
  3. Implement deterministic verifiers checking both final scientific accuracy and intermediate tool outputs.
  4. Run zero-shot evaluations comparing Sage S2.1 Flash with and without Sage Sci-Perception inputs.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

Papers such as SciAgentGym and MatToolBench focus on multi-step scientific tool-use and benchmarking, but none of the listed works examine containerized end-to-end evaluation combining native structured inputs and software tools using the Harbor format with full-interface vs text-only ablated baselines.

We searched OpenAlex and arXiv for “multimodal scientific agent benchmark molecular spectral tool use” and “End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic Workflows” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Sage S2.1 Flash Technical Report

Kun Cai, Haodong Chen, Guangran Cheng +203 more · 2026

Sage S2.1 Flash is a 320-billion-parameter mixture-of-experts (MoE) model designed to integrate native scientific perception with general-purpose multimodal reasoning and long-horizon agentic execution. It combines dedicated scientific encoders and modality-specific projectors (Sage Sci-Perception) with post-training across large-scale verifiable environments spanning tool use, search, coding, ML engineering, workspace deliverables, and computer use, achieving top-two rankings on many of 33 evaluated benchmarks.

Related Ideas

Is this a good idea?

Share onXLinkedIn