Apexin AI · 2026-08-18 · notable
ASI-Bench — 60 research projects that remove the method step by step
ASI-Bench gives an agent a real research project, then takes the method away in four stages. Across 18 agent-model setups the average score falls from 50.91 with full guidance to 26.62 when the agent must pick its own method.
ASI-Bench measures how far an AI agent gets on a real research project once you stop telling it which method to use.
Key specs
| Tasks | 60 |
|---|---|
| Domains | 11 |
| Expert hours | 31,000+ |
What is it?
ASI-Bench is a benchmark of 60 project-level research tasks across 11 scientific domains, including astronomy, chemistry, materials, mathematics, medicine and robotics. More than 40 experts spent over 31,000 hours building it. Every task exists at four guidance levels, so the same project can be handed to an agent with the full procedure spelled out or with almost nothing but the goal.
How does it work?
The four levels withdraw help in steps. B1 gives background, method, equations and the whole procedure. B2 names the method and its constraints but leaves the agent to build a workflow. B3 gives only the objective, data, constraints and required outputs, so the agent chooses the method itself. B4 adds true but irrelevant information on top of B3 to test whether the agent gets distracted. Tasks are checked by expert review, AI-assisted auditing, sandbox execution and scorer validation.
Why does it matter?
Scores on ASI-Bench collapse as guidance is removed: 50.91 at B1, 29.10 at B2, 26.62 at B3 and 26.99 at B4, averaged over 18 agent-model configurations. That gap is a concrete measure of how much of an agent's apparent research skill is really the human method it was handed. The runner and task data are public, so a team can reproduce the numbers rather than take them on trust.
Who is it for?
agent evaluation and AI-for-science researchers
Try it
pip install asibench && asibench task pull --repo seed31415