smevals

// Overview

smevals is a framework for running evals against small (and large) models. It organises an evaluation into four pieces: an eval is the high-level capability you care about, tasks are the individual exercises inside it, configs describe the model setups you want to compare, and graders hold the assessment logic. Splitting grading from running is the point — you can re-grade an existing set of results with new criteria without paying to run the tasks again.

Runs are immutable records with timestamps and full provenance, and the runner and checker outputs are preserved as artefacts so a surprising score can be traced back to what the model actually produced. Checks are individual assertions or measurements, either built in or your own scripts, and several graders can coexist on one eval.

The workflow is four commands: run the tasks, grade the runs, generate a markdown report, or serve a web UI to browse the results — leaderboards, metrics and tag aggregations. It is a Python package installed with uv or pip, released under the MIT licence, and was built by Simon Willison with Jesse Vincent's Prime Radiant lab.

// What it does

• Four-part structure — evals, tasks, configs and graders — that separates what you run from how you score it
• Immutable runs with timestamps and provenance, so a result can always be traced back
• Multiple graders per eval, applied to existing runs without re-running the tasks
• Built-in checks plus custom scripts for assertions and measurements
• Runner and checker artefacts preserved for later analysis
• Markdown reports and a web UI with leaderboards, metrics and tag aggregations

// Getting started

smevals is a Python CLI. Install it, run an eval, then grade and report on the results.

uv tool install smevals
# or: pip install smevals

smevals run EVAL

smevals grade EVAL
smevals report EVAL

smevals serve EVAL

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Compare a small open model against a frontier one on a task you actually care about
• Re-score an expensive set of runs with a new grader instead of re-running them
• Keep an auditable history of how a model scored on your own suite over time
• Share a results dashboard with a team without building reporting yourself

// How smevals compares

smevals alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.