# simple-evals

> OpenAI's lightweight harness for running standard zero-shot, chain-of-thought LLM benchmarks

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: MIT
- Repository: https://github.com/openai/simple-evals
- Canonical page: https://ai-tldr.dev/tools/simple-evals/

## In short

simple-evals is a small Python library from OpenAI for evaluating language models on standard benchmarks. It was open sourced so the accuracy numbers OpenAI publishes alongside its models can be reproduced and inspected. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed MIT.

## Overview

simple-evals is a small Python library from OpenAI for evaluating language models on standard benchmarks. It was open sourced so the accuracy numbers OpenAI publishes alongside its models can be reproduced and inspected. The emphasis is on a zero-shot, chain-of-thought setup, which the maintainers consider closer to how models are actually used than many-shot prompting.

It targets people who need to measure model accuracy on known academic tests, such as researchers comparing models or engineers checking a model against a published score. Out of the box it covers benchmarks like MMLU, MATH, GPQA, HumanEval, MGSM, DROP, and SimpleQA, and it ships adapters for the OpenAI and Anthropic APIs.

As a benchmark harness, it sits in the evaluation and testing category: you point it at a model name, pick a benchmark and a number of examples, and it produces accuracy numbers. Note that the repo is in maintenance mode and, as of July 2025, is no longer updated for new models or results, though it continues to host reference implementations for HealthBench, BrowseComp, and SimpleQA.

## What it does

- Reference implementations of common benchmarks: MMLU, MATH, GPQA, HumanEval, MGSM, DROP, and SimpleQA
- Zero-shot, chain-of-thought prompting setup used for OpenAI's published accuracy numbers
- Built-in adapters for the OpenAI and Anthropic APIs via the `openai` and `anthropic` packages
- Simple CLI: list available models, then run an eval by model name and example count
- Configurable number of examples per run, useful for quick smoke tests or full runs
- Published benchmark tables you can compare your own runs against

## Getting started

simple-evals has no unified installer because dependencies are optional; install only what you need, set your API key, then run an eval from the command line.

1. Install the API client you need

   Install the OpenAI client, the Anthropic client, or both, depending on which models you want to evaluate.

```bash
pip install openai
pip install anthropic
```

2. Add HumanEval support (optional)

   The HumanEval benchmark needs the separate human-eval package.

```bash
git clone https://github.com/openai/human-eval
pip install -e human-eval
```

3. Set your API key

   Export the relevant *_API_KEY environment variable before running, for example OPENAI_API_KEY or ANTHROPIC_API_KEY.

```bash
export OPENAI_API_KEY=sk-...
```

4. List models and run an eval

   List the available model names, then run a benchmark by passing a model and the number of examples.

```bash
python -m simple-evals.simple_evals --list-models
python -m simple-evals.simple_evals --model <model_name> --examples <num_examples>
```

## When to use it

- Reproduce or sanity-check OpenAI's published accuracy numbers for a given model
- Compare two models on benchmarks like MMLU, MATH, or GPQA under the same zero-shot setup
- Run a quick smoke test on a small number of examples before a full benchmark run
- Use the reference implementations of SimpleQA, HealthBench, or BrowseComp as a starting point for your own evals