agent-eval

// Overview

agent-eval, from Hugging Face's `is-it-agentic-enough` repository, asks a question ordinary code benchmarks skip: not "did the agent finish the task" but "how did it use your library to get there". It runs the same task set across models and library revisions and records the shape of the interaction — whether the agent reached for the CLI or hand-wrote Python, how many tokens it burned, how long it took, and how often it errored.

Tasks are run under three access tiers so you can see what documentation is worth. `bare` gives the agent nothing beyond the model's own knowledge, `clone` puts the repository in the working directory, and `skill` hands it a packaged Skill. Comparing the tiers on the same tasks is the experiment: if the skill tier does not move the numbers, the skill is not earning its place.

A run is described by a YAML matrix — profile, tasks, number of runs, hardware flavor, models and library revisions — and `agent-eval batch` launches it as Hugging Face Jobs. Results come back as a static HTML report with match percentage, median time, median tokens, error rate, label adoption (CLI versus `pipeline()`), per-run distributions and a task-by-revision coverage heatmap. Traces can be synced to a Hugging Face bucket or packaged into a dataset. It is Apache-2.0 and targets Python 3.13.

// What it does

• Measures token cost, wall-clock time, error rate and usage style alongside plain correctness
• Three access tiers — bare, clone and skill — so you can quantify what your docs or Skill are worth
• YAML matrix over models × library revisions, launched as Hugging Face Jobs
• Static HTML reports with per-run distributions and a task × revision coverage heatmap
• Trace sync to a Hugging Face bucket and packaging of runs into a dataset

// Getting started

agent-eval installs from source with uv. You supply the tasks and a YAML matrix describing what to run.

git clone https://github.com/huggingface/is-it-agentic-enough
cd is-it-agentic-enough
uv venv --python 3.13 .env
uv pip install --python .env/bin/python -e .

agent-eval batch matrix.yaml

agent-eval report <profile>
agent-eval sync
agent-eval upload <repo>

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Find out whether a coding agent can drive your library at all before you write more docs for it
• Compare library revisions to see whether an API change made agents faster or more error-prone
• Decide if a packaged Skill earns its maintenance by comparing the skill tier against bare and clone
• Compare models on the same library rather than on a generic coding benchmark

// How agent-eval compares

agent-eval alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.