Simon Willison · 2026-07-31 · notable
smevals — Simon Willison and Prime Radiant ship a small eval suite
smevals is a small Python eval framework Simon Willison built with Jesse Vincent's Prime Radiant lab. It runs YAML-defined tasks across model/prompt/harness configs, then graders score the outputs and a web app browses the results.
A small, opinionated eval suite from Simon Willison and Jesse Vincent's Prime Radiant lab for testing models, prompts, and harnesses.
What is it?
smevals is a Python CLI that runs eval suites defined as directories of YAML configuration plus executable scripts. Simon Willison and Jesse Vincent's Prime Radiant lab built the framework, and Simon calls this his third iteration on the shape — 'it feels right to me'. Each eval bundles tasks that measure a specific capability such as text-to-SQL, spec-satisfaction, or 'draw a pelican riding a bicycle'.
How does it work?
The framework has a small vocabulary: evals contain tasks; runs execute those tasks against a chosen model or model-plus-harness config; graders then score the outputs, with checks and checkers as the underlying assertions. smevals deliberately separates running from grading, so a run can be regraded later without re-invoking the model. A built-in web app browses the leaderboard of runs and grades.
Why does it matter?
Public eval frameworks target big benchmarks. smevals fills the inner loop most builders actually need: quickly compare two prompts, two harnesses, or two model versions on a task you care about, without wiring a bespoke harness each time. It comes from a trusted voice with a track record of small, sharp Python tools, and MIT-licensed on GitHub.
Try it
uvx smevals docs