MirroS Lab · 2026-08-17 · notable
HarnessEval-W — agents score world models and show their work
HarnessEval-W is an open-source benchmark that grades video world models with AI agents instead of fixed metrics. MirroS Lab ran it over 18 world models and 330 cases, and every score ships with the reasoning trace behind it.
An agent-run benchmark for world models that returns an evidence trail, not just a number.
Key specs
| World models evaluated | 18 |
|---|---|
| Evaluation cases | 330 |
| Human rank correlation | rho = 0.93 |
What is it?
HarnessEval-W replaces the fixed scoring rubric with a panel of evaluation agents. MirroS Lab published the benchmark on 17 August 2026 together with results for 18 world models across 330 evaluation cases. Each case produces an auditable case card that records how the verdict was reached, and the repository ships a bundled example so you can run the pipeline end to end before plugging in your own model.
How does it work?
The pipeline reads each evaluation case, splits the question into measurable sub-questions, and routes them to specialist sub-agents that reason over the evidence. Those agents build an evidence tree across 11 skills grouped under quality, transitions and persistence. HarnessEval-W scored 5,940 rollouts this way, keeping the full reasoning chain for every one of them.
Why does it matter?
A single scalar score hides why a generated clip failed, which makes world-model comparisons hard to trust. On intentional transitions the rankings from HarnessEval-W match human rankings at rho = 0.93, and on physical tasks it agrees with human pairwise choices 71.7% of the time. Researchers get a diagnosis they can check rather than a leaderboard they have to take on faith.
Who is it for?
world-model and video generation researchers
Try it
git clone https://github.com/MirroS-Lab/HarnessEval-W