OpenMOSS · 2026-08-20 · notable
SWE-bench Science — the best coding agent fixes under half of 119 science bugs
SWE-bench Science is an MIT-licensed benchmark of 119 repository-level tasks drawn from 98 scientific codebases across 20 domains. On the published leaderboard, Claude Opus 5 leads with 47.90% Pass@1.
119 real bug-fixing tasks pulled from 98 scientific codebases, where every agent tested fails more than half.
Key specs
| Tasks | 119 |
|---|---|
| Repositories | 98 |
| Best pass@1 | 47.90% |
What is it?
SWE-bench Science tests coding agents on software that scientists actually run: 119 repository-level tasks across 98 GitHub repositories and 20 scientific domains, covering density-functional theory, plasma stability, genomics, molecular dynamics and computational physics. Python dominates, with C++, MATLAB/Octave and Fortran also present. OpenMOSS released the dataset on Hugging Face under MIT.
How does it work?
Tasks are split into three paradigms — Issue-driven, Expert-exploratory and Engineering-integration — and Pass@1 is reported for each as well as overall. Every task ships Docker container specs for both environment setup and verification, plus the repository URL, commit hash and source license, so a fix only counts when the scientific result checks out.
Why does it matter?
The leaderboard puts Claude Opus 5 (max) first at 47.90% Pass@1, GPT-5.6 Sol (max) at 46.22% and DeepSeek V4 Pro (max) at 42.02%, so more than half of these bugs go unfixed by every agent tested. The paper names four repeat failures: knowledge gaps, surface-level repairs, incomplete integration and poor generalization. An ablation found that scientific guidance helps only when it is well aligned, and hurts when it is not.
Who is it for?
researchers pointing coding agents at scientific code
Try it
Load OpenMOSS-Team/SWE-bench-Science from the Hugging Face datasets hub