# agent-eval

> Measure how well coding agents actually use your library

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/huggingface/is-it-agentic-enough
- Canonical page: https://ai-tldr.dev/tools/agent-eval/
- Updated: 2026-06-18

## In short

agent-eval is Measure how well coding agents actually use your library. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0. To get started: install.

## Overview

agent-eval, from Hugging Face's `is-it-agentic-enough` repository, asks a question ordinary code benchmarks skip: not "did the agent finish the task" but "how did it use your library to get there". It runs the same task set across models and library revisions and records the shape of the interaction — whether the agent reached for the CLI or hand-wrote Python, how many tokens it burned, how long it took, and how often it errored.

Tasks are run under three access tiers so you can see what documentation is worth. `bare` gives the agent nothing beyond the model's own knowledge, `clone` puts the repository in the working directory, and `skill` hands it a packaged Skill. Comparing the tiers on the same tasks is the experiment: if the skill tier does not move the numbers, the skill is not earning its place.

A run is described by a YAML matrix — profile, tasks, number of runs, hardware flavor, models and library revisions — and `agent-eval batch` launches it as Hugging Face Jobs. Results come back as a static HTML report with match percentage, median time, median tokens, error rate, label adoption (CLI versus `pipeline()`), per-run distributions and a task-by-revision coverage heatmap. Traces can be synced to a Hugging Face bucket or packaged into a dataset. It is Apache-2.0 and targets Python 3.13.

## What it does

- Measures token cost, wall-clock time, error rate and usage style alongside plain correctness
- Three access tiers — bare, clone and skill — so you can quantify what your docs or Skill are worth
- YAML matrix over models × library revisions, launched as Hugging Face Jobs
- Static HTML reports with per-run distributions and a task × revision coverage heatmap
- Trace sync to a Hugging Face bucket and packaging of runs into a dataset

## Getting started

agent-eval installs from source with uv. You supply the tasks and a YAML matrix describing what to run.

1. Install

   Clone the repo and install it into a Python 3.13 environment.

```bash
git clone https://github.com/huggingface/is-it-agentic-enough
cd is-it-agentic-enough
uv venv --python 3.13 .env
uv pip install --python .env/bin/python -e .
```

2. Launch a matrix

   The YAML file names the profile, tasks, run count, hardware flavor, models and library revisions to sweep.

```bash
agent-eval batch matrix.yaml
```

3. Build the report and keep the traces

   `report` renders the static HTML; `sync` mirrors results to or from a Hugging Face bucket and `upload` packages the traces as a dataset.

```bash
agent-eval report <profile>
agent-eval sync
agent-eval upload <repo>
```

## When to use it

- Find out whether a coding agent can drive your library at all before you write more docs for it
- Compare library revisions to see whether an API change made agents faster or more error-prone
- Decide if a packaged Skill earns its maintenance by comparing the skill tier against bare and clone
- Compare models on the same library rather than on a generic coding benchmark

## Version history

- 2026-06-18: Hugging Face published agent-eval and the accompanying write-up, measuring coding agents on library use across the bare, clone and skill access tiers rather than task completion alone.