What is ClawBench?
ClawBench is an open-source benchmark and evaluation harness for AI browser agents, built by TIGER-AI-Lab and the NAIL Group. Instead of sandboxed replicas, its tasks run on live consumer websites — ordering food, booking travel, applying for jobs, writing reviews, managing projects. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.
Overview
ClawBench is an open-source benchmark and evaluation harness for AI browser agents, built by TIGER-AI-Lab and the NAIL Group. Instead of sandboxed replicas, its tasks run on live consumer websites — ordering food, booking travel, applying for jobs, writing reviews, managing projects. The repository ships two corpora: V1 with 152 tasks across 143 live sites and V2 with 129 tasks across 63 sites, plus a 20-task ClawBench-Lite subset of V1 for cheaper iteration. The accompanying paper is arXiv 2604.08523, accepted to EMNLP 2026 Findings.
Each run spins up an isolated Docker or Podman container with Chromium and the selected agent harness. A request interceptor attached over Chrome DevTools Protocol blocks critical, irreversible HTTP requests — checkout, form submit, email send — so nothing real is purchased or sent, and at that moment five layers of recording are frozen: a session video, action screenshots, browser actions, HTTP traffic and the agent's messages. The agent works as a synthetic user (Alex Green) with a personal-info file, a resume and a fresh disposable email address per run.
Scoring for V2 and newer leaderboard rows is two-stage: a deterministic check that the intercepted request matches the task's URL and method schema, then an LLM judge that checks the captured payload against the natural-language instruction under a lenient and a strict rubric. The model and the agent scaffold are separate axes — the harness flag swaps between OpenClaw (the default), Hermes Agent, opencode, Claude Code, OpenAI Codex CLI, browser-use, claw-code, Pi and baselines such as random-click and null. Task definitions and full execution traces are published as Hugging Face datasets, and the scores live on the leaderboard at claw-bench.com.
What it does
- Everyday tasks on live websites: 152 V1 tasks over 143 sites and 129 V2 tasks over 63 sites in 15 life categories, plus a 20-task Lite subset
- Docker- or Podman-isolated Chromium per run, with a noVNC URL to watch the agent operate in real time
- A CDP request interceptor that blocks irreversible submits and freezes five recording layers: video, screenshots, actions, HTTP traffic and agent messages
- Interchangeable agent harnesses — OpenClaw, Hermes Agent, opencode, Claude Code, Codex CLI, browser-use, claw-code, Pi — plus random-click and null baselines
- Two-stage scoring (interception match, then an LLM judge with lenient and strict rubrics), re-scoring of saved runs and a batch analyzer for failure taxonomy and per-category results
- Human mode for recording reference runs, managed remote browsers (Kernel, Browserbase) and a Harbor-framework path for V2
Getting started
ClawBench is published on PyPI as `clawbench-eval`. You need Python 3.11+, uv and a container engine — Docker or Podman, auto-detected (force one with `CONTAINER_ENGINE`). The first run builds a container image with Chromium, ffmpeg, noVNC and the harness dependencies; later runs reuse the cached layers.
Install the CLI
This installs the `clawbench`, `clawbench-run`, `clawbench-batch`, `clawbench-rescore`, `clawbench-analyze`, `clawbench-reproduce` and `clawbench-harbor-adapt` commands. `pipx install clawbench-eval` works too; clone the repo and run `./run.sh` instead if you want to modify the driver, the bundled test cases or the container build.
uv tool install clawbench-evalConfigure models and the judge
Run `clawbench` from the directory where results should live; on first launch it creates templates under `models/`. Add your agent models plus an API key for the judge model — the README uses `deepseek-v4-pro`, the judge behind every published leaderboard row.
clawbench
$EDITOR models/models.yamlRun one task
From a source checkout, run a single test case against one configured model. The script prints a noVNC URL (e.g. http://localhost:6080/vnc.html) to watch the agent live; results land in `./test-output/<model>/…` with the full five-layer recording. Running `clawbench` with no arguments opens an interactive TUI with guided model and task selection.
uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6Run a whole corpus
`--cases-suite v2` runs the full V2 corpus (use `v1-lite` for the 20-task subset). Add `--max-concurrent N` for parallel tasks, `--harness <name>` to pick an agent scaffold, or `--no-judge` to skip scoring. A `batch-summary.json` plus per-run recordings are written under `./test-output/`.
clawbench-batch --models your-model --cases-suite v2 --all-casesAnalyze or re-score
Summarise a finished batch's Stage-1/Stage-2 rates, failure taxonomy and per-category results, or re-judge saved runs with both rubrics.
clawbench-analyze --runs-dir test-output/batch-<timestamp> --out report.md
clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric bothCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Measure whether a browser or computer-use agent actually finishes everyday online tasks on real production sites, not on a frozen sandbox
- Compare agent scaffolds with the model held fixed (or models with the scaffold held fixed) using the `--harness` axis
- Debug agent failures from the five-layer trace — video, screenshots, DOM actions, HTTP requests and the agent transcript
- Reproduce or re-judge a published leaderboard row from the open trace datasets without re-running the agent
How ClawBench compares
ClawBench alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| LM Evaluation Harness | ★ 14.4k | EleutherAI's framework for few-shot evaluation of language models across 60+ academic benchmarks, used as the backend for many leaderboards. |
| OpenCompass | ★ 7.6k | An LLM evaluation platform that runs models against 100+ datasets covering reasoning, knowledge, coding, and domain tasks, with leaderboards and multi-model support. |
| SWE-bench | ★ 6.1k | A benchmark and containerized harness that tests whether language models can resolve real GitHub issues by generating patches that pass a repository's tests. |
| simple-evals | ★ 4.7k | OpenAI's lightweight library for running standard zero-shot, chain-of-thought benchmarks like MMLU, MATH, and GPQA to measure model accuracy. |
| lmms-eval | ★ 4.5k | An evaluation suite for large multimodal models that runs image, video, and audio benchmarks across many tasks with a unified, reproducible interface. |
| AgentBench | ★ 3.8k | A benchmark that evaluates LLMs as agents across diverse interactive environments such as operating systems, databases, web browsing, and games. |
| EvalScope | ★ 3.6k | ModelScope evaluation framework that runs standard benchmarks against local or OpenAI-compatible models, with agent-loop evaluation, inference stress testing and a comparison dashboard. |
| ClawBench | ★ 1.2k | Run browser agents on everyday tasks across live websites, recorded and judged |