# AgentBench

> Benchmark LLMs as agents across operating systems, databases, web, and games

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/THUDM/AgentBench
- Website: https://llmbench.ai/agent
- Canonical page: https://ai-tldr.dev/tools/agentbench/

## In short

AgentBench is a benchmark for measuring how well large language models behave as autonomous agents, not just as text generators. Instead of single-shot questions, it puts a model into multi-turn interactive environments and scores whether it can reach a goal. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

AgentBench is a benchmark for measuring how well large language models behave as autonomous agents, not just as text generators. Instead of single-shot questions, it puts a model into multi-turn interactive environments and scores whether it can reach a goal. The original release covers 8 distinct environments, including freshly built ones such as Operating System, Database, Knowledge Graph, Digital Card Game, and Lateral Thinking Puzzles, plus tasks recompiled from ALFWorld, WebShop, and Mind2Web.

The project is aimed at researchers and engineers who build or compare agentic LLMs and want a repeatable way to test tool use, planning, and grounded interaction. The current main branch is AgentBench FC (Function Calling), which uses a function-calling style prompt and is integrated with the AgentRL framework. It ships fully containerized deployment for five tasks: alfworld (AF), dbbench (DB), knowledgegraph (KG), os_interaction (OS), and webshop (WS).

As a benchmark harness in the evaluation category, AgentBench focuses on running agents against fixed task suites and reporting scores on a public leaderboard. Older versions (v0.1 and v0.2) remain available as tags if you need the earlier task set or the non-Docker conda workflow.

## What it does

- Evaluates LLMs as agents across diverse interactive environments rather than static QA
- Original suite spans 8 environments: OS, DB, KG, Digital Card Game, Lateral Thinking Puzzles, plus ALFWorld, WebShop, and Mind2Web
- AgentBench FC adds a function-calling prompt style integrated with the AgentRL RL framework
- Fully containerized deployment for alfworld, dbbench, knowledgegraph, os_interaction, and webshop via Docker Compose
- One-command setup brings up task workers, a controller, a Freebase server, and Redis for container allocation
- Public leaderboard and Dev/Test splits for reporting and comparing model results

## Getting started

AgentBench FC (the current main branch) runs its task environments in containers via Docker Compose. The steps below follow the README's one-command setup.

1. Get the code

   Clone the repository and enter the directory.

```bash
git clone https://github.com/THUDM/AgentBench.git
cd AgentBench
```

2. Pull and build the task images

   Some tasks need prebuilt Docker images. Pull MySQL for dbbench and build the OS interaction images.

```bash
# dbbench
docker pull mysql:8

# os_interaction
docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles
docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles
docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles
```

3. Bring up the stack

   Start the controller and task workers with Docker Compose. The webshop environment needs about 16GB of RAM, so make sure your machine has enough resources.

```bash
docker compose -f extra/docker-compose.yml up
```

## When to use it

- Compare how different LLMs perform as agents on the same fixed set of interactive tasks
- Stress-test a model's tool use and multi-turn planning in OS, database, and web environments
- Track agent capability over model versions and report results to the public leaderboard
- Provide task environments for training and evaluating function-calling agents alongside AgentRL