Skip to content

Source paper

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

Wanghan Xu, Yuhao Zhou, Yifan Zhou and 104 more

Wanghan Xu, Yuhao Zhou, Yifan Zhou, Qinglong Cao, Shuo Li, Jia Bu, Bo Liu, Yixin Chen, Xuming He, Xiangyu Zhao, Xiang Zhuang, Fengxiang Wang, Zhiwang Zhou, Qiantai Feng, Wenxuan Huang, Jiaqi Wei, Hao Wu, Yuejin Yang, Guangshuai Wang, Sheng Xu, Ziyan Huang, Xinyao Liu, Jiyao Liu, Cheng Tang, Wei Li, Ying Chen, Junzhi Ning, Pengfei Jiang, Chenglong Ma, Ye Du, Changkai Ji, Huihui Xu, Ming Hu, Jiangbin Zheng, Xin Chen, Yucheng Wu, Feifei Jiang, Xi Chen, Xiangru Tang, Yuchen Fu, Yingzhou Lu, Yuanyuan Zhang, Lihao Sun, Chengbo Li, Jinzhe Ma, Wanhao Liu, Yating Liu, Kuo-Cheng Wu, Shengdu Chai, Yizhou Wang, Ouwen Zhangjin, Chen Tang, Shufei Zhang, Wenbo Cao, Junjie Ren, Taoyong Cui, Zhouheng Yao, Juntao Deng, Yijie Sun, Feng Liu, Wangxu Wei, Jingyi Xu, Zhangrui Li, Junchao Gong, Zijie Guo, Zhiyu Yao, Zaoyu Chen, Tianhao Peng, Fangchen Yu, Bo Zhang, Dongzhan Zhou, Shixiang Tang, Jiaheng Liu, Fenghua Ling, Yan Lu, Yuchen Ren, Ben Fei, Zhen Zhao, Xinyu Gu, Rui Su, Xiao-Ming Wu, Weikang Si, Yang Liu, Hao Chen, Xiangchao Yan, Xue Yang, Junchi Yan, Jiamin Wu, Qihao Zheng, Chenhui Li, Zhiqiang Gao, Hao Kong, Junjun He, Mao Su, Tianfan Fu, Peng Ye, Chunfeng Song, Nanqing Dong, Yuqiang Li, Huazhu Fu, Siqi Sun, Lijing Cheng, Jintai Lin, Wanli Ouyang, Bowen Zhou, Wenlong Zhang, Lei Bai

arXiv 2512.16969 · 202586 upvotes on Hugging Face

Research ideas from this paper (5)

Generated by AI. The paper’s authors did not write these.

Meta-Analytic Agents with Verification-Aware Aggregation

Deploy meta-analytic agents that extract retrieved scientific data into structured dataframes and use code interpreters to execute precise numerical aggregations.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
🟡Fairly easy💡Moderate InnovationSingle GPU or lessClosely matching work found#Agents#AI for science

Constraint-Grounded Feasibility Verifiers for Idea Generation

Implement a verifiable feasibility critic trained on technical protocols to parameterize generated research ideas and reject plans that violate resource constraints.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
🟠Medium🔬Significant ResearchSingle GPU or lessClosely matching work found#Reasoning#AI for science

Numerical Stability-Aware RL for Dry Experiments

Fine-tune coding models using reinforcement learning with rewards based on numerical error minimization against ground-truth simulations to prevent cascading errors from unstable approximations.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
🔴Hard🔬Significant ResearchLarge clusterNo close match found#Reinforcement learning#AI for science

Graph-Based Branching Protocols for Wet Lab Automation

Generate experimental workflows as directed acyclic graphs instead of linear text to accurately capture complex branching logic and parallel execution in wet lab automation.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
⛔Very hard🚀GroundbreakingSingle GPU or lessNo close match found#Agents#AI for science

Contrastive Tuning for Fine-Grained Comparative Reasoning

Apply contrastive fine-tuning on a curated dataset of spot-the-difference scientific figure pairs to train models in distinguishing subtle quantitative visual shifts.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
🟠Medium💡Moderate InnovationLarge clusterNo close match found#Reasoning#Multimodal

AI summary

This paper introduces Scientific General Intelligence (SGI), a framework grounded in the Practical Inquiry Model (PIM) that evaluates AI on the complete cycle of scientific discovery: Deliberation, Conception, Action, and Perception. To measure this, the authors introduce SGI-Bench, a scientist-aligned benchmark comprising over 1,000 expert-curated samples across 10 disciplines, revealing that while current LLMs are linguistically fluent, they lack the integrated reasoning capabilities required for genuine scientific discovery.

How it works (AI summary)

The study operationalizes SGI through four distinct tasks: Scientific Deep Research (literature review/meta-analysis), Idea Generation (hypothesis formation), Dry/Wet Experiments (code and protocol design), and Experimental Reasoning (multimodal interpretation). Evaluation is conducted using an agentic framework (Agent-as-a-judge) with multi-dimensional metrics, and the authors also explore Test-Time Reinforcement Learning (TTRL) to improve hypothesis novelty.