# Claw-Eval

> 300 human-verified agent tasks graded on the whole trajectory

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: MIT
- Repository: https://github.com/claw-eval/claw-eval
- Website: https://claw-eval.github.io
- Canonical page: https://ai-tldr.dev/tools/claw-eval/
- Updated: 2026-04-07

## In short

Claw-Eval is a benchmark and evaluation harness for autonomous agents. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed MIT. To get started: create the environment.

## Overview

Claw-Eval is a benchmark and evaluation harness for autonomous agents. It ships 300 human-verified tasks with 2,159 rubrics across nine categories, split into `general` (161 tasks covering communication, finance, ops and productivity), `multimodal` (101 tasks on webpage generation, video QA and document extraction) and `multi_turn` (38 conversational tasks driven by simulated user personas).

What separates it from output-only benchmarks is that agents are graded on three dimensions through full-trajectory auditing: completion (did the agent finish the task), safety (did it avoid harmful or unauthorised actions), and robustness (does it pass consistently). The primary metric is Pass^3 — a model only earns credit if it meets the success criteria in all three of three independent trials — which is designed to eliminate lucky runs.

The harness is a Python CLI you point at a model config; `claw-eval batch` runs a config with sandboxing, a trial count and a parallelism setting. The dataset is published on Hugging Face and ModelScope, the leaderboard is live at claw-eval.github.io, and EvalScope integrates Claw-Eval so you can run it against any OpenAI-compatible endpoint with standardised reports. The paper is arXiv 2604.06132.

## What it does

- 300 human-verified tasks with 2,159 rubrics across nine categories and three splits (general, multimodal, multi-turn)
- Full-trajectory auditing — execution traces and environment state, not just the final answer
- Three grading dimensions: completion, safety and robustness
- Pass^3 as the primary metric: a task counts only when the model passes all three independent trials
- Sandboxed batch runner with per-config task selection and configurable parallelism
- Dataset published on Hugging Face and ModelScope, with a public leaderboard and an EvalScope integration for OpenAI-compatible endpoints

## Getting started

The README recommends uv for dependency management and Python 3.11. You will need an OpenRouter key, plus a search key for the tasks that hit the real web.

1. Create the environment

   uv gives a fast, reproducible virtualenv.

```bash
pip install uv
uv venv --python 3.11
source .venv/bin/activate
```

2. Set keys and build the sandbox

   One script prepares the environments. `SERP_DEV_KEY` is only needed for tasks that require real web search.

```bash
export OPENROUTER_API_KEY=sk-or-...
export SERP_DEV_KEY=...
bash scripts/test_sandbox.sh
```

3. Fetch the full fixtures

   File-size limits keep the complete fixtures (including videos) off GitHub — download them from the Hugging Face dataset `claw-eval/Claw-Eval`; the smaller set lives in `data/fixtures.tar.gz`.

4. Run a model

   Point the runner at a model config and give it three trials for the Pass^3 metric. Separate configs exist for the general, multimodal and user-agent splits.

```bash
claw-eval batch --config model_configs/claude_opus_46.yaml --sandbox --trials 3 --parallel 16
```

5. Or drive it from EvalScope

   EvalScope supports running Claw-Eval against OpenAI-compatible endpoints with CLI/Python usage, saved predictions and standardised reports — see evalscope.readthedocs.io/en/latest/benchmarks/claw_eval.html.

## When to use it

- Measure whether an agent finished a task safely and repeatably, rather than whether one run produced the right final answer
- Compare foundation models on general agentic capability using a shared, human-verified rubric set
- Regression-test your own agent stack across three trials before shipping a prompt or tool change
- Reproduce or contest a public leaderboard result — the codebase and dataset are open and the protocol is documented

## Version history

- 2026-04-07: Covered on AI/TLDR alongside the arXiv paper: a 300-task benchmark that grades agents on the full trajectory — execution traces, audit logs and environment snapshots — catching 44% of safety violations that final-output grading misses.