Simon Willison · 2026-07-16 · notable
Simon Willison — Kimi K3 and the pelican benchmark come apart
Simon Willison runs Moonshot's new 2.8T Kimi K3 through his 'pelican riding a bicycle' SVG test — the drawing cost about 25 cents and burned 13,241 reasoning tokens, and he argues the pelican has stopped tracking real model strength.
Simon Willison retires the pelican benchmark as a comparative metric — GLM-5.2 draws a better one than models that beat it everywhere else.
What is it?
Simon Willison's 2026-07-16 post walks through what happened when he pointed his usual pelican-riding-a-bicycle prompt at Kimi K3, Moonshot's new 2.8T MoE model. Kimi K3's SVG rendered cleanly but cost roughly 25 cents and consumed 13,241 reasoning tokens at the only reasoning-effort level Moonshot currently exposes.
How does it work?
The post treats the pelican as three separate signals bundled into one image. First — capability check: can the model produce a valid SVG at all? Second — cost estimation: how many reasoning tokens does it burn on a small drawing? Third — comparative ranking, which is where the pelican breaks down: GLM-5.2's pelican looks better than Kimi K3's, Claude Fable 5's, and GPT-5.6 Sol's, so the ranking no longer matches the models' scores on real benchmarks.
Why does it matter?
The pelican was probably the most-cited informal LLM benchmark on X, so Willison retiring it as a comparative metric is a signal for anyone still using aesthetic vibes-checks to grade frontier models. He argues the test is still useful as a reproducible 'hello world' — a sanity check that a new model can produce structured output and an easy way to eyeball reasoning-token cost — just not as a leaderboard.
Who is it for?
ML practitioners and voices who track frontier-model evaluation
Try it
Read: simonwillison.net/2026/Jul/16/kimi-k3/