Overview
Open ASR Leaderboard is the evaluation code behind Hugging Face's public speech-recognition leaderboard. It runs automatic speech recognition models over a shared set of public benchmarks on the Hub and reports word error rate alongside RTFx, the inverse real-time factor that captures how fast a model transcribes. The leaderboard itself is a Gradio Space; this repository is what produces the numbers on it, which means anyone can reproduce a published result or score a model that is not listed yet.
The benchmark set is deliberately broad: English short-form test sets, a long-form suite built from earnings calls (with CORAAL kept separate because it has multiple splits), and a multilingual suite spanning FLEURS, Common Voice and MLS. A chunked variant splits long calls into segments and concatenates a call's predictions before scoring against the call-level reference, so long-form scores stay comparable across models with different context limits. Models submitted to the leaderboard are additionally evaluated by the maintainers on private sets.
Reproducibility is enforced by the execution model rather than by convention. English and multilingual short-form evaluations run on Hugging Face Jobs: every run executes a Docker image on the same hardware flavour, so environment and driver differences do not leak into the score, and each model family has its own image and eval configuration published as a Space. Each family exposes a `run_eval.py` entry point driven by a shell script that writes predictions to JSONL and prints WER and RTFx, so contributors can evaluate locally before Jobs support lands for a suite. Plotting scripts turn the results into the accuracy-versus-speed trade-off charts the leaderboard shows, with any model highlighted.
What it does
- One scoring pipeline and one text normalizer across every ASR model family, so WER and RTFx are comparable
- English short-form, long-form (earnings21/22, CORAAL) and multilingual (FLEURS, Common Voice, MLS) benchmark suites
- Chunked long-form evaluation that concatenates a call's chunk predictions before scoring against a call-level reference
- Hugging Face Jobs execution — the same Docker image on the same hardware flavour for every run, with per-family eval configs published as Spaces
- Local `run_eval.py` entry point per model family for contributors testing before submission
- Trade-off plotting scripts (RTFx versus WER) with per-model highlighting, plus support for API-served models
Getting started
Launching an evaluation needs a Hugging Face account with Jobs credits, a WRITE token and a storage bucket for results; the local setup only launches jobs and scores with the repo's normalizer, so no GPU is required on your machine.
Clone and create the launcher environment
A minimal conda environment is enough to submit jobs and score results locally.
git clone git@github.com:huggingface/open_asr_leaderboard.git
cd open_asr_leaderboard
conda create -n leaderboard_jobs python=3.10 -y
conda activate leaderboard_jobs
pip install -r requirements/requirements_jobs.txt
huggingface-cli loginSubmit an evaluation
Open the model family's submit script, uncomment the models and datasets you want, then run it. Jobs are submitted one per dataset in parallel; the script waits for them, syncs results from your bucket and prints a CSV summary.
RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash qwen/submit_jobs.shScore a long-form or multilingual model locally
Suites that have not moved to Jobs yet run from the per-family requirements files; each family's bash script drives run_eval.py, writes predictions to JSONL and prints WER and RTFx.
pip install -r requirements/requirements_whisper.txt
bash run_whisper.shPlot the accuracy/speed trade-off
Turns the CSV results into the RTFx-versus-WER charts used on the leaderboard, optionally highlighting one model.
./scripts/plot_all.sh --highlight "nvidia/parakeet-tdt-0.6b-v3"Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Reproduce a leaderboard result instead of taking a published WER on trust
- Benchmark your own or an unlisted ASR model against the same datasets and normalizer everyone else is scored with
- Compare accuracy against throughput before picking a speech model for production
- Evaluate long-form and multilingual performance, not just clean English short-form audio
How Open ASR Leaderboard compares
Open ASR Leaderboard alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| LM Evaluation Harness | ★ 14.1k | EleutherAI's framework for few-shot evaluation of language models across 60+ academic benchmarks, used as the backend for many leaderboards. |
| OpenCompass | ★ 7.5k | An LLM evaluation platform that runs models against 100+ datasets covering reasoning, knowledge, coding, and domain tasks, with leaderboards and multi-model support. |
| SWE-bench | ★ 5.9k | A benchmark and containerized harness that tests whether language models can resolve real GitHub issues by generating patches that pass a repository's tests. |
| simple-evals | ★ 4.6k | OpenAI's lightweight library for running standard zero-shot, chain-of-thought benchmarks like MMLU, MATH, and GPQA to measure model accuracy. |
| lmms-eval | ★ 4.4k | An evaluation suite for large multimodal models that runs image, video, and audio benchmarks across many tasks with a unified, reproducible interface. |
| AgentBench | ★ 3.7k | A benchmark that evaluates LLMs as agents across diverse interactive environments such as operating systems, databases, web browsing, and games. |
| EvalScope | ★ 3.5k | ModelScope evaluation framework that runs standard benchmarks against local or OpenAI-compatible models, with agent-loop evaluation, inference stress testing and a comparison dashboard. |
| Open ASR Leaderboard | ★ 255 | Hugging Face's evaluation code for speech-recognition models — one scoring pipeline across English, long-form and multilingual ASR benchmarks |