A group of 20 researchers has released SCI-AGENT-EXAM, a benchmark that tests whether AI agents can do realistic scientific research work. In a paper posted to arXiv on Oct. 23, they report that the best agent tested, OpenAI’s GPT-6 Astra, averaged a score of 80.9 but cleared their success bar in only 26.0% of runs.
The paper’s lead author is Tianyun Zhong. The authors list affiliations with the University of Chinese Academy of Sciences, the Chinese Academy of Sciences’ Institute of Software, Alibaba’s AI Data group and Qwen team, Zhejiang University, the Hong Kong University of Science and Technology and Tsinghua University.
What the tasks look like
The benchmark has 120 tasks written by experts across 11 scientific fields. The authors sort them into seven types of research work, from literature synthesis and hypothesis validation to code generation, error diagnosis and interpreting results.
Each task asks for two kinds of output, the authors wrote: working artifacts such as code and scripts, and written reports that explain the findings. That reflects how research actually gets done, they argue, since existing benchmarks often test only one slice of scientific work.
Many tasks contain a trap.
“The obvious method, the loudest signal, or a ready-made result is engineered to be wrong, and credit requires seeing past it,” the paper says.
In one data science task, the workspace includes a file formatted like a finished answer, with a verdict to advance a drug biomarker screen. An agent that copies it fails. An agent that rebuilds the analysis from the source article finds the screen doesn’t hold up.
Grading uses two tracks. Code and numbers are checked by programs that rerun the work, sometimes on altered data so hard-coded answers fail. Reports are scored by an AI judge that the authors calibrated against human experts.
The results
The team tested 10 frontier models, run through the Claude Code harness.
GPT-6 Astra scored 80.9 overall. Claude Fable 5 came second at 67.3, followed by Claude Opus 5 at 64.2, Qwen3.8 Max at 62.7 and Kimi K3 at 62.6. DeepSeek-V4-Pro finished last at 50.9.
The success threshold is a normalized score of at least 0.95. No other model got there in more than 10.5% of runs, the rate for Claude Opus 5.
Strengths varied by field. Astra led in 10 of the 11 domains, the paper says. Claude Fable 5 scored 80 in engineering but 49 in bioinformatics, and Kimi K3 scored 81 in materials science but 50 in physical science.
The two grading tracks often disagreed.
The share of cases with a strong report but weak code or numbers ranged from 0.14 to 0.48 across models, the authors found.
In one example, Claude Opus 5 passed all 23 report criteria and named the right answer, but its supporting calculations didn’t reproduce. “The conclusion is right and the narrative is airtight,” the authors wrote, but the evidence behind it fails.
Cost per run also varied. Astra cost $20.79 per run, while GLM-5.3 cost the most at $57.96, averaging 102.4 turns. The authors say that shows efficiency should be judged per run, not by token prices alone.
The paper notes its limits. It tests general-purpose models in standard agent settings, not specialized science agents with their own tools and memory. The code and data are public on GitHub.
Source: arXiv, Tianyun Zhong et al., “SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks”
Additional sources: arXiv (HTML full text), Tianyun Zhong et al., “SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks, full paper”; GitHub, SKYLENAGE-AI, “Sci-Agent-Exam code and data”
