Dan Luu · 2026-09-07 · notable
Dan Luu — telling a coding agent to use a test technique barely helps
Dan Luu ran 26 testing instructions and 4 agent skills against the same Rust Zstd task, 80 runs each with Codex on GPT-5.6 Sol. Giving no special instruction at all scored well above average.
26 testing instructions, 4 skills, 80 runs each — and none of them clearly beat giving the agent no instruction at all.
What is it?
Dan Luu tested whether telling a coding agent to use a specific test technique makes its code more correct. The task was implementing Zstd in Rust, run under 26 prompt conditions — TDD, Lean 4, Verus, Kani, QuickCheck, Proptest, fuzzing, mutation testing, TLA+, SMT solvers and more — plus 4 agent skills, at 80 runs per condition per effort level with Codex on GPT-5.6 Sol. He also repeated it on the IMAP RFC at 40 runs per condition.
How does it work?
The score is the fraction of runs that pass 100% of hidden tests, plotted against cost. Reading what the agents actually did explains the flat result: given a named technique, they either write their normal tests inside that technique's framework, or use the technique so shallowly that its value never arrives. Property-based testing turned into random inputs hammering rejection cases; formal methods turned into proofs of irrelevant properties. Agents could point at risky code but still failed to write tests that hit it.
Why does it matter?
The ranking matters less than the baseline: 'Default', meaning no extra instruction, landed well above average, and the popular pre-made test skills Codex itself recommended underperformed — including one from a skill collection with 250,000 GitHub stars. Luu's own quick custom skill did fine, and he credits the difference to nudging agents away from their default behaviour rather than reading like a tutorial. His open question for labs: why has nobody built RL environments that teach agents to test well?
Who is it for?
engineers relying on coding agents to write their tests