# OpenAI Evals

> A framework and open registry for evaluating LLMs and LLM-based systems

- Category: Evaluation & Production Ops / Evaluation & Red-Teaming
- Language: Python
- Licence: MIT
- Repository: https://github.com/openai/evals
- Canonical page: https://ai-tldr.dev/tools/openai-evals/

## In short

OpenAI Evals is a framework for evaluating large language models (LLMs) and systems built on top of them. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed MIT. To get started: set your OpenAI API key.

## Overview

OpenAI Evals is a framework for evaluating large language models (LLMs) and systems built on top of them. It ships with an open registry of existing evals that test different dimensions of OpenAI models, and it lets you write your own custom evals for the use cases you care about.

It is aimed at developers building with LLMs who need a repeatable way to measure how different model versions affect their application. Many basic and model-graded evals require no evaluation code at all: you provide your data in JSON and set parameters in YAML. You can also build private evals against your own data without exposing it publicly.

Within the evaluation and testing space, Evals acts as a standard harness for running and comparing model behaviour. It supports more advanced setups, such as prompt chains and tool-using agents, through its Completion Function Protocol.

## What it does

- Open registry of ready-made evals covering many dimensions of model behaviour
- Write custom evals, including model-graded evals defined in YAML
- Build private evals against your own data without publishing it
- Eval templates that need no evaluation code, just JSON data plus YAML parameters
- Completion Function Protocol for prompt chains and tool-using agents
- Optional logging of results to a Snowflake database or Weights & Biases

## Getting started

You need Python 3.9 or newer and an OpenAI API key. Set the key once, then install the package depending on whether you want to run existing evals or create your own.

1. Set your OpenAI API key

   Evals call the OpenAI API, so export your key as an environment variable before running anything. Be aware of the API usage costs.

```bash
export OPENAI_API_KEY=your-api-key
```

2. Install to run existing evals

   If you only want to run evals locally rather than contribute new ones, install the package from PyPI.

```bash
pip install evals
```

3. Install from source to create evals

   If you plan to write evals, clone the repo and install in editable mode so your changes take effect without reinstalling.

```bash
pip install -e .
```

4. Fetch registry data

   The evals registry is stored with Git-LFS. After installing LFS, pull the data files from within your local copy of the repo.

```bash
cd evals
git lfs fetch --all
git lfs pull
```

## When to use it

- Measure how a new model version changes the behaviour of your LLM application before you ship it
- Run existing registry evals to compare models across different tasks
- Build a private eval on your own production data without exposing it publicly
- Evaluate prompt chains or tool-using agents with the Completion Function Protocol