EvalEval Coalition / Hugging Face · 2026-04-29 · major
Hugging Face: AI Evals Are Becoming the New Compute Bottleneck — $40K for One Agent Leaderboard Run
An EvalEval Coalition analysis published on Hugging Face finds agent benchmark costs have crossed a structural threshold: HAL costs $40K per run, GAIA runs $2,829 per model, and reliability testing multiplies those figures 8x — pricing academics out of independent evaluation.
Evaluation costs now rival training costs — and they can't be compressed for agents, threatening who gets to define AI capability.
What is it?
The EvalEval Coalition published this analysis on Hugging Face on April 29, 2026, documenting how the economics of AI evaluation have fundamentally shifted. While training costs have always been a barrier, evaluation costs for modern agent benchmarks now match or exceed training at a governance-threatening scale.
How does it work?
The analysis compiles real costs across major benchmarks: HAL spends $40K for 21,730 agent rollouts; a single GAIA frontier-model run costs $2,829; PaperBench costs ~$9,500 per evaluation. Static benchmark compression (100-200x reduction) works, but agent benchmarks only compress 2-3.5x before rank fidelity breaks. Training-in-the-loop benchmarks have no general compression solution. Adding 8-run reliability testing multiplies all costs by ~8x.
Why does it matter?
Whoever can afford evaluation writes the leaderboard. Cost-blind rankings reward wasteful inference strategies, and academic groups cannot independently replicate frontier agent results. The paper calls for standardized eval data sharing via the Every Eval Ever schema, cost-reporting alongside accuracy on leaderboards, and evaluation budget allocation in research funding — reframing the problem as a governance issue, not a solvable optimization.