Does a low word error rate still mean an ASR model is accurate?

Not on its own, according to the Benchmark Fitting analysis. Hugging Face and Hume AI report that the highest-scoring models were the most likely to reproduce erroneous VoxPopuli transcripts, so part of a strong word error rate reflects learned test-set patterns. The new tab is meant to be read next to word error rate, not instead of it.

Are the ASR test sets themselves clean?

Not fully. The Benchmark Fitting team found that 40% of VoxPopuli test clips contain potential reference errors, touching about 3% of all reference words. Artificial Analysis published a cleaned version as VoxPopuli-Cleaned-AA, and the reference-disagreement probe uses those known errors to measure how often a model repeats them.

Did the models behave differently on freshly recorded audio?

Yes, and that is the control the study leans on. The Benchmark Fitting write-up reports weaker benchmark-optimization behaviour on newly collected audio from the same domains — European Parliament and LibriVox recordings made in 2026 — which separates genuine domain skill from memorization of the published test sets.

How can the Benchmark Fitting results be reproduced?

The evaluation scripts sit under benchmark_fitting in the Apache-2.0 huggingface/open_asr_leaderboard repository, and Hume AI published its own code at HumeAI/asr-benchmark-optimization. Hugging Face also released the un-normalized model outputs in a public bucket, so the scoring can be rechecked without re-running all eleven speech models.

Hugging Face · 2026-08-21 · major

Benchmark Fitting — Hugging Face scores how much ASR models memorize tests

Hugging Face and Hume AI added a Benchmark Fitting tab to the Open ASR Leaderboard. Three probes check whether a speech model transcribes the audio or repeats the reference. Six of 11 models dropped audible words the reference lacked.

Thumbnail for the Hugging Face post on measuring benchmark optimization in speech recognition

A new leaderboard tab measures how much a speech model memorized the test set instead of hearing the audio.

Quick facts

Maker	Hugging Face with Hume AI, Artificial Analysis and Treble Technologies
What's new	Benchmark Fitting tab on the Open ASR Leaderboard
Probes	Reference disagreement, masked entity retrieval, orthographic switching
Models tested	11, including Whisper large-v3, Parakeet, Canary, Qwen3-ASR and Voxtral
Also launched	Real World VoiceEQ and a far-field ASR leaderboard
License	Apache-2.0 (evaluation scripts)
Paper	Towards Quantifying Benchmark Optimization in ASR Models

What is it?

The Open ASR Leaderboard now carries a Benchmark Fitting tab, built by Hugging Face with Hume AI, Artificial Analysis and Treble Technologies. Per model, it reports how much of a good word error rate comes from learned test-set patterns rather than transcription skill. Two further boards launched alongside it: Real World VoiceEQ and a far-field ASR leaderboard.

How does it work?

Three behavioural probes drive the score. Reference disagreement checks whether a model repeats a known transcript error instead of the words actually spoken. Masked entity retrieval silences a number in the audio and sees whether the model fills it back in — some models did so 30–40% of the time on LibriSpeech. Orthographic switching tests whether a model copies each dataset's house style, matching conventions such as "Mr." against "Mister" up to 90% of the time.

Why does it matter?

Word error rate is how most teams pick a speech vendor, and these results argue that leading scores are partly a test-taking skill: six of eleven models dropped audible phrases when the audio contradicted the benchmark reference. Anyone comparing ASR systems now has a second column that estimates how much of the headline number survives contact with their own recordings.

Who is it for?

speech ML engineers and teams choosing an ASR vendor

Frequently asked questions

Does a low word error rate still mean an ASR model is accurate?: Not on its own, according to the Benchmark Fitting analysis. Hugging Face and Hume AI report that the highest-scoring models were the most likely to reproduce erroneous VoxPopuli transcripts, so part of a strong word error rate reflects learned test-set patterns. The new tab is meant to be read next to word error rate, not instead of it.
Are the ASR test sets themselves clean?: Not fully. The Benchmark Fitting team found that 40% of VoxPopuli test clips contain potential reference errors, touching about 3% of all reference words. Artificial Analysis published a cleaned version as VoxPopuli-Cleaned-AA, and the reference-disagreement probe uses those known errors to measure how often a model repeats them.
Did the models behave differently on freshly recorded audio?: Yes, and that is the control the study leans on. The Benchmark Fitting write-up reports weaker benchmark-optimization behaviour on newly collected audio from the same domains — European Parliament and LibriVox recordings made in 2026 — which separates genuine domain skill from memorization of the published test sets.
How can the Benchmark Fitting results be reproduced?: The evaluation scripts sit under benchmark_fitting in the Apache-2.0 huggingface/open_asr_leaderboard repository, and Hume AI published its own code at HumeAI/asr-benchmark-optimization. Hugging Face also released the un-normalized model outputs in a public bucket, so the scoring can be rechecked without re-running all eleven speech models.

Try it

https://huggingface.co/spaces/hf-audio/open_asr_leaderboard