Inspect AI

// Overview

Inspect is a Python framework for evaluating large language models, built by the UK AI Security Institute (AISI). You define an evaluation as a task that pairs a dataset with a solver (how the model is prompted or run) and a scorer (how its answers are judged), then run it against any supported model from the command line or Python.

It is aimed at researchers and engineers who need repeatable, structured evals rather than one-off prompt tests. The framework ships built-in components for prompt engineering, tool use, multi-turn dialog, and model-graded scoring, and can be extended by other Python packages to add new elicitation or scoring techniques.

As an eval framework, Inspect gives you a common harness for measuring model behavior across providers. It also includes a collection of over 200 pre-built evaluations you can run on any model, so you can start from existing benchmarks instead of writing everything from scratch.

// What it does

• Tasks composed of a dataset, a solver, and a scorer, defined in plain Python with the @task decorator
• Built-in support for prompt engineering, tool usage, and multi-turn dialog
• Model-graded scoring (for example model_graded_fact) for judging open-ended answers
• Over 200 pre-built evaluations ready to run on any supported model
• Run evals from the CLI with inspect eval against models from different providers
• Extensible through other Python packages for new scoring and elicitation techniques

// Getting started

Install the package, define a task that pairs a dataset, solver, and scorer, then run it from the command line against your chosen model.

pip install inspect-ai

from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate

@task
def security_guide():
    return Task(
        dataset=example_dataset("security_guide"),
        solver=generate(),
        scorer=model_graded_fact()
    )

inspect eval security_guide.py --model openai/gpt-4o

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Measuring how a model performs on a benchmark by running one of the 200+ pre-built evaluations
• Building a custom eval for your own task with a dataset, solver, and model-graded scorer
• Comparing models across providers on the same task using a single harness
• Testing tool use and multi-turn agent behavior in a structured, repeatable way

// How Inspect AI compares

Inspect AI alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.