# ClawBench > Run browser agents on everyday tasks across live websites, recorded and judged - Category: Evaluation & Production Ops / Benchmark Harnesses - Language: Python - Licence: Apache-2.0 - Repository: https://github.com/TIGER-AI-Lab/ClawBench - Website: https://claw-bench.com - Canonical page: https://ai-tldr.dev/tools/clawbench/ - Updated: 2026-10-05 ## In short ClawBench is an open-source benchmark and evaluation harness for AI browser agents, built by TIGER-AI-Lab and the NAIL Group. Instead of sandboxed replicas, its tasks run on live consumer websites — ordering food, booking travel, applying for jobs, writing reviews, managing projects. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0. ## Overview ClawBench is an open-source benchmark and evaluation harness for AI browser agents, built by TIGER-AI-Lab and the NAIL Group. Instead of sandboxed replicas, its tasks run on live consumer websites — ordering food, booking travel, applying for jobs, writing reviews, managing projects. The repository ships two corpora: V1 with 152 tasks across 143 live sites and V2 with 129 tasks across 63 sites, plus a 20-task ClawBench-Lite subset of V1 for cheaper iteration. The accompanying paper is arXiv 2604.08523, accepted to EMNLP 2026 Findings. Each run spins up an isolated Docker or Podman container with Chromium and the selected agent harness. A request interceptor attached over Chrome DevTools Protocol blocks critical, irreversible HTTP requests — checkout, form submit, email send — so nothing real is purchased or sent, and at that moment five layers of recording are frozen: a session video, action screenshots, browser actions, HTTP traffic and the agent's messages. The agent works as a synthetic user (Alex Green) with a personal-info file, a resume and a fresh disposable email address per run. Scoring for V2 and newer leaderboard rows is two-stage: a deterministic check that the intercepted request matches the task's URL and method schema, then an LLM judge that checks the captured payload against the natural-language instruction under a lenient and a strict rubric. The model and the agent scaffold are separate axes — the harness flag swaps between OpenClaw (the default), Hermes Agent, opencode, Claude Code, OpenAI Codex CLI, browser-use, claw-code, Pi and baselines such as random-click and null. Task definitions and full execution traces are published as Hugging Face datasets, and the scores live on the leaderboard at claw-bench.com. ## What it does - Everyday tasks on live websites: 152 V1 tasks over 143 sites and 129 V2 tasks over 63 sites in 15 life categories, plus a 20-task Lite subset - Docker- or Podman-isolated Chromium per run, with a noVNC URL to watch the agent operate in real time - A CDP request interceptor that blocks irreversible submits and freezes five recording layers: video, screenshots, actions, HTTP traffic and agent messages - Interchangeable agent harnesses — OpenClaw, Hermes Agent, opencode, Claude Code, Codex CLI, browser-use, claw-code, Pi — plus random-click and null baselines - Two-stage scoring (interception match, then an LLM judge with lenient and strict rubrics), re-scoring of saved runs and a batch analyzer for failure taxonomy and per-category results - Human mode for recording reference runs, managed remote browsers (Kernel, Browserbase) and a Harbor-framework path for V2 ## Getting started ClawBench is published on PyPI as `clawbench-eval`. You need Python 3.11+, uv and a container engine — Docker or Podman, auto-detected (force one with `CONTAINER_ENGINE`). The first run builds a container image with Chromium, ffmpeg, noVNC and the harness dependencies; later runs reuse the cached layers. 1. Install the CLI This installs the `clawbench`, `clawbench-run`, `clawbench-batch`, `clawbench-rescore`, `clawbench-analyze`, `clawbench-reproduce` and `clawbench-harbor-adapt` commands. `pipx install clawbench-eval` works too; clone the repo and run `./run.sh` instead if you want to modify the driver, the bundled test cases or the container build. ```bash uv tool install clawbench-eval ``` 2. Configure models and the judge Run `clawbench` from the directory where results should live; on first launch it creates templates under `models/`. Add your agent models plus an API key for the judge model — the README uses `deepseek-v4-pro`, the judge behind every published leaderboard row. ```bash clawbench $EDITOR models/models.yaml ``` 3. Run one task From a source checkout, run a single test case against one configured model. The script prints a noVNC URL (e.g. http://localhost:6080/vnc.html) to watch the agent live; results land in `./test-output//…` with the full five-layer recording. Running `clawbench` with no arguments opens an interactive TUI with guided model and task selection. ```bash uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6 ``` 4. Run a whole corpus `--cases-suite v2` runs the full V2 corpus (use `v1-lite` for the 20-task subset). Add `--max-concurrent N` for parallel tasks, `--harness ` to pick an agent scaffold, or `--no-judge` to skip scoring. A `batch-summary.json` plus per-run recordings are written under `./test-output/`. ```bash clawbench-batch --models your-model --cases-suite v2 --all-cases ``` 5. Analyze or re-score Summarise a finished batch's Stage-1/Stage-2 rates, failure taxonomy and per-category results, or re-judge saved runs with both rubrics. ```bash clawbench-analyze --runs-dir test-output/batch- --out report.md clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric both ``` ## When to use it - Measure whether a browser or computer-use agent actually finishes everyday online tasks on real production sites, not on a frozen sandbox - Compare agent scaffolds with the model held fixed (or models with the scaffold held fixed) using the `--harness` axis - Debug agent failures from the five-layer trace — video, screenshots, DOM actions, HTTP requests and the agent transcript - Reproduce or re-judge a published leaderboard row from the open trace datasets without re-running the agent ## Version history - 2026-10-05 v0.11.0: Runs now record code, corpus and agent revisions in run-meta.json; a judge outage gets its own outcome, container waits are bounded so one wedged run can't stall a batch, and Harbor gains a Kernel browser arm and an optional OpenEval/EvalPort export script.