Overview
Claw-Eval is a benchmark and evaluation harness for autonomous agents. It ships 300 human-verified tasks with 2,159 rubrics across nine categories, split into `general` (161 tasks covering communication, finance, ops and productivity), `multimodal` (101 tasks on webpage generation, video QA and document extraction) and `multi_turn` (38 conversational tasks driven by simulated user personas).
What separates it from output-only benchmarks is that agents are graded on three dimensions through full-trajectory auditing: completion (did the agent finish the task), safety (did it avoid harmful or unauthorised actions), and robustness (does it pass consistently). The primary metric is Pass^3 — a model only earns credit if it meets the success criteria in all three of three independent trials — which is designed to eliminate lucky runs.
The harness is a Python CLI you point at a model config; `claw-eval batch` runs a config with sandboxing, a trial count and a parallelism setting. The dataset is published on Hugging Face and ModelScope, the leaderboard is live at claw-eval.github.io, and EvalScope integrates Claw-Eval so you can run it against any OpenAI-compatible endpoint with standardised reports. The paper is arXiv 2604.06132.
What it does
- 300 human-verified tasks with 2,159 rubrics across nine categories and three splits (general, multimodal, multi-turn)
- Full-trajectory auditing — execution traces and environment state, not just the final answer
- Three grading dimensions: completion, safety and robustness
- Pass^3 as the primary metric: a task counts only when the model passes all three independent trials
- Sandboxed batch runner with per-config task selection and configurable parallelism
- Dataset published on Hugging Face and ModelScope, with a public leaderboard and an EvalScope integration for OpenAI-compatible endpoints
Getting started
The README recommends uv for dependency management and Python 3.11. You will need an OpenRouter key, plus a search key for the tasks that hit the real web.
Create the environment
uv gives a fast, reproducible virtualenv.
pip install uv
uv venv --python 3.11
source .venv/bin/activateSet keys and build the sandbox
One script prepares the environments. `SERP_DEV_KEY` is only needed for tasks that require real web search.
export OPENROUTER_API_KEY=sk-or-...
export SERP_DEV_KEY=...
bash scripts/test_sandbox.shFetch the full fixtures
File-size limits keep the complete fixtures (including videos) off GitHub — download them from the Hugging Face dataset `claw-eval/Claw-Eval`; the smaller set lives in `data/fixtures.tar.gz`.
Run a model
Point the runner at a model config and give it three trials for the Pass^3 metric. Separate configs exist for the general, multimodal and user-agent splits.
claw-eval batch --config model_configs/claude_opus_46.yaml --sandbox --trials 3 --parallel 16Or drive it from EvalScope
EvalScope supports running Claw-Eval against OpenAI-compatible endpoints with CLI/Python usage, saved predictions and standardised reports — see evalscope.readthedocs.io/en/latest/benchmarks/claw_eval.html.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Measure whether an agent finished a task safely and repeatably, rather than whether one run produced the right final answer
- Compare foundation models on general agentic capability using a shared, human-verified rubric set
- Regression-test your own agent stack across three trials before shipping a prompt or tool change
- Reproduce or contest a public leaderboard result — the codebase and dataset are open and the protocol is documented
How Claw-Eval compares
Claw-Eval alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| LM Evaluation Harness | ★ 14k | EleutherAI's framework for few-shot evaluation of language models across 60+ academic benchmarks, used as the backend for many leaderboards. |
| OpenCompass | ★ 7.5k | An LLM evaluation platform that runs models against 100+ datasets covering reasoning, knowledge, coding, and domain tasks, with leaderboards and multi-model support. |
| SWE-bench | ★ 5.9k | A benchmark and containerized harness that tests whether language models can resolve real GitHub issues by generating patches that pass a repository's tests. |
| simple-evals | ★ 4.6k | OpenAI's lightweight library for running standard zero-shot, chain-of-thought benchmarks like MMLU, MATH, and GPQA to measure model accuracy. |
| lmms-eval | ★ 4.4k | An evaluation suite for large multimodal models that runs image, video, and audio benchmarks across many tasks with a unified, reproducible interface. |
| AgentBench | ★ 3.7k | A benchmark that evaluates LLMs as agents across diverse interactive environments such as operating systems, databases, web browsing, and games. |
| HELM | ★ 2.9k | Stanford CRFM's Holistic Evaluation of Language Models framework for reproducible, transparent benchmarking of foundation and multimodal models across many scenarios and metrics. |
| Claw-Eval | ★ 773 | 300 human-verified agent tasks graded on the whole trajectory |