Overview
HarnessEval-W is an evaluation pipeline for visual world models from MirroS Lab. Its argument is that a benchmark should deliver more than a scalar: judging a video rollout means deciding whether physics, causality and world state evolve correctly, and a brute-force metric leaves no reasoning chain anyone can examine. HarnessEval instead brings the harness paradigm from the LLM ecosystem to world-model benchmarking, so each result arrives with the evidence that produced it.
The pipeline runs in three stages. A planner routes each case — an initial world, an action and a probe intent — to the skills that can legitimately evaluate it, recording an evidence-grounded reason for every skill it skips. Each selected skill then decomposes its evaluation into measurable sub-questions, and a dedicated sub-agent answers each one against the rollout evidence. Finally the parent agent validates the collected evidence and aggregates it into the case score. Routing depends only on the case context and never on the model under test, so every model faces the same questions on the same cases.
The repository ships the benchmark, the evaluation code and the metric backends, with a bundled demo under runs/example that can be scored immediately after setup. A completed run writes summary.json with overall and per-family scores alongside leaderboard files in JSON, CSV and Markdown, with per-case artifacts and metric caches kept under the run directory. A 100-case public set and a leaderboard are published on Hugging Face and the project page, and the benchmark is designed to be extended with new cases and new evaluation skills. The code is Apache-2.0; the accompanying paper is arXiv 2608.16859.
What it does
- Agentic evaluation: a planner routes cases to skills, sub-agents answer measurable sub-questions, and a parent agent validates and aggregates
- Every score is backed by an auditable case card — the full trace of questions, answers, scores and supporting frames
- Skipped skills are recorded with an evidence-grounded reason, not silently dropped
- Routing depends only on case context, never on the model under test, so models are compared on identical questions
- Bundled demo run that produces scores immediately after installation, plus verification of a completed run
- Outputs summary.json with overall and per-family scores plus leaderboard files in JSON, CSV and Markdown
- Extensible with new cases (world, action, probe family) and new evaluation skills
Getting started
Installation creates three conda environments — the launcher/CLI, the metric backends and the physical-plausibility backend — then you configure credentials and can score the bundled demo straight away.
Clone and create the environments
Three separate environments keep the CLI, the metric backends and the physical-plausibility backend from fighting over dependencies.
git clone --branch main --single-branch https://github.com/mirros-lab/harnesseval-w.git
cd harnesseval-w
conda env create -f docs/installation/main.environment.yml
conda env create -f docs/installation/metrics.environment.yml
conda env create -f docs/installation/pavrm.environment.yml
conda activate harnesseval-mainConfigure credentials and paths
Copy the example env file and export it into your shell.
cp config/example.env harnesseval.env
set -a; . ./harnesseval.env; set +aEvaluate a model's generated results
Point the CLI at the generated videos plus the run manifest and plan root; it writes the run outputs under --run-root.
harnesseval eval \
--results runs/example/results_example/generation \
--model-id seedance-2.0-standard \
--run-root runs/example/results_example/run \
--manifest runs/example/results_example/manifest.json \
--plan-root benchmark/plansVerify a completed run
Checks that an existing run has all the expected scores.
harnesseval verify run \
--eval-root runs/example/results_example/run/harnesseval/models/seedance-2.0-standard/evaluation \
--manifest runs/example/results_example/manifest.json \
--model seedance-2.0-standardRead the outputs
A completed evaluation writes overall and per-family scores plus leaderboard files; per-case artifacts and metric caches sit alongside them.
runs/example/results_example/run/harnesseval/models/<model-id>/evaluation/
├── summary.json
├── leaderboard_latest.json
├── leaderboard_latest.csv
└── LEADERBOARD.mdCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Score a video world model on physics, causality and world-state consistency instead of a single opaque quality metric
- Audit why a model lost points on a case by reading the evidence tree behind the score
- Compare several world models on identical routed questions, with per-family breakdowns and a generated leaderboard
- Extend a benchmark with your own cases or evaluation skills rather than writing a harness from scratch
How HarnessEval-W compares
HarnessEval-W alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| LM Evaluation Harness | ★ 14k | EleutherAI's framework for few-shot evaluation of language models across 60+ academic benchmarks, used as the backend for many leaderboards. |
| OpenCompass | ★ 7.5k | An LLM evaluation platform that runs models against 100+ datasets covering reasoning, knowledge, coding, and domain tasks, with leaderboards and multi-model support. |
| SWE-bench | ★ 5.9k | A benchmark and containerized harness that tests whether language models can resolve real GitHub issues by generating patches that pass a repository's tests. |
| simple-evals | ★ 4.6k | OpenAI's lightweight library for running standard zero-shot, chain-of-thought benchmarks like MMLU, MATH, and GPQA to measure model accuracy. |
| lmms-eval | ★ 4.4k | An evaluation suite for large multimodal models that runs image, video, and audio benchmarks across many tasks with a unified, reproducible interface. |
| AgentBench | ★ 3.7k | A benchmark that evaluates LLMs as agents across diverse interactive environments such as operating systems, databases, web browsing, and games. |
| HELM | ★ 2.9k | Stanford CRFM's Holistic Evaluation of Language Models framework for reproducible, transparent benchmarking of foundation and multimodal models across many scenarios and metrics. |
| HarnessEval-W | ★ 297 | Agentified evaluation of visual world models, where every score comes with its evidence tree |