Reinvently · 2026-08-23 · notable
Ed-o-meter — a 28-task LLM leaderboard that runs every model down one track
Ed-o-meter scores 17 models on 28 practical tasks across coding, data, real-world work, security and tool use, showing pass rate, cost and time-to-first-token side by side. GLM-5.3 passes all five categories for $0.28 a run.
One set of prompts, one API path, 17 models — a small independent leaderboard that puts cost and latency next to the pass rate.
Key specs
| Models | 17 |
|---|---|
| Tasks | 28 |
What is it?
Ed-o-meter is a public LLM leaderboard built by Ed Yau that runs 17 models through 28 hand-written tasks and publishes the result of each one. The tasks are grouped into coding, data, real-world, security and tool use. Every model gets the same prompts, the same automated pass/fail checkers and the same OpenRouter streaming path, so the cost and latency columns can be compared directly.
How does it work?
Scoring mixes an automated pass/fail check per task with a rubric score out of 10 judged afterwards by an LLM, and the page describes the setup as "same driver, same track" — serial time trials with nothing else running. Refusals are recorded rather than hidden, and latency is measured as time to first token. The harness underneath, Featherbench, is a single-file MIT-licensed Python project on GitHub, so a team can fork it and swap in its own tasks.
Why does it matter?
The August 23 update put GLM-5.3 on top with a 100% pass rate, a 9.3 rubric score and $0.28 for a full run. GPT-5.5 also passed everything, at 8.8 rubric and $1.43 a run, and was quickest at 13.2 seconds to first token, while Kimi K3 earned the best rubric score at 9.5 but was slowest at 26.4 seconds. Putting quality, price and speed on one row makes the trade-off easy to see before you commit to a model.
Who is it for?
engineers picking a model for production work
Try it
https://github.com/ed-is-ai/featherbench