# Terminal-Bench

> A benchmark and harness for testing AI agents on real terminal tasks

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Repository: https://github.com/harbor-framework/terminal-bench
- Website: https://www.tbench.ai
- Canonical page: https://ai-tldr.dev/tools/terminal-bench/

## In short

Terminal-Bench is a benchmark for testing how well AI agents handle real terminal tasks, from compiling code to setting up servers and training models. It has two parts: a dataset of tasks and an execution harness that connects a language model to a sandboxed terminal environment. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python.

## Overview

Terminal-Bench is a benchmark for testing how well AI agents handle real terminal tasks, from compiling code to setting up servers and training models. It has two parts: a dataset of tasks and an execution harness that connects a language model to a sandboxed terminal environment.

It is built for people working on LLM agents, benchmarking frameworks, or system-level reasoning. Each task ships with an English instruction, a test script that checks whether the agent succeeded, and a reference "oracle" solution. You run everything through the tb command-line tool.

As an evaluation and benchmark harness, it gives you a reproducible task suite and a runner so you can score agents the same way each time. It is currently in beta with around 100 tasks, and there is a public leaderboard you can submit to.

## What it does

- Two-part design: a dataset of terminal tasks plus an execution harness that runs them
- Each task includes an English instruction, a verification test script, and a reference oracle solution
- Runs agents against a sandboxed terminal environment using Docker
- Single CLI (tb) to run evaluations, with flags for agent, model, dataset name, and version
- Versioned datasets (e.g. terminal-bench-core v0.1.1) tied to a public leaderboard
- Open to contributions of new tasks and benchmark adapters

## Getting started

Terminal-Bench ships as a pip package and is driven by the tb CLI. You also need uv and Docker installed to run the harness.

1. Install the package

   Install Terminal-Bench with uv (recommended) or pip.

```bash
uv tool install terminal-bench
```

2. Install with pip (alternative)

   If you prefer pip, install the same package directly.

```bash
pip install terminal-bench
```

3. See the harness options

   The harness connects a model to a sandboxed terminal. View the available run options with the help flag.

```bash
tb run --help
```

4. Run against the leaderboard dataset

   Evaluate an agent and model on Terminal-Bench-Core. Pass the dataset name and version to match the current leaderboard.

```bash
tb run \
    --agent terminus \
    --model anthropic/claude-3-7-latest \
    --dataset-name terminal-bench-core \
    --dataset-version 0.1.1 \
    --n-concurrent 8
```

## When to use it

- Benchmark how well an LLM agent completes real, end-to-end command-line tasks
- Compare different agents or models on the same reproducible task suite
- Stress-test an agent's system-level reasoning in a sandboxed shell before shipping
- Submit results to the Terminal-Bench leaderboard or contribute new tasks and adapters