WhisperX

// Overview

WhisperX is a speech-recognition pipeline built around OpenAI's Whisper model. Plain Whisper produces accurate text but only gives timestamps at the utterance level, and those can be off by several seconds; it also has no native batching. WhisperX fixes both problems by wrapping Whisper in a four-stage pipeline — voice-activity detection, batched inference, forced phoneme alignment, and speaker diarization.

The alignment stage is what gives WhisperX its name recognition. After Whisper produces a transcript, a phoneme-based model in the wav2vec2 family is used to align the orthographic text back onto the audio, which yields a start and end time for every individual word rather than for each whole segment. A separate diarization stage, using pyannote-audio, partitions the audio by speaker and attaches a speaker label to each segment.

Because inference runs through the faster-whisper backend with VAD-based batching, the project reports roughly 70x realtime transcription using Whisper large-v2 while fitting in under 8 GB of GPU memory at beam size 5. WhisperX was accepted at INTERSPEECH 2023 and took first place in the Ego4d transcription challenge.

// What it does

• Batched inference on the faster-whisper backend — the project reports ~70x realtime with Whisper large-v2
• Word-level timestamps produced by forced phoneme alignment against wav2vec2 models
• Multi-speaker transcription: speaker diarization via pyannote-audio attaches speaker IDs to segments
• VAD preprocessing that trims silence, reduces hallucination, and enables batching without degrading WER
• Sentence-level transcript segments (via nltk sentence tokenization) for cleaner subtitles
• Command-line entry point plus subtitle output, including a --highlight_words flag that visualises word timings in the .srt

// Getting started

WhisperX ships on PyPI. GPU acceleration needs the CUDA toolkit installed first; CPU-only use can skip that step. Speaker diarization additionally requires a Hugging Face read token and acceptance of the pyannote model's user agreement.

pip install whisperx

whisperx path/to/audio.wav

whisperx path/to/audio.wav --hf_token YOUR_HF_READ_TOKEN

git clone https://github.com/m-bain/whisperX.git
cd whisperX
uv sync --all-extras --dev

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Generate subtitle files where each word is timed precisely enough to karaoke-highlight during playback
• Transcribe multi-speaker recordings — interviews, panels, meetings — with a speaker label on every segment
• Batch-process a large archive of audio on a single GPU, where plain Whisper's lack of batching makes the job too slow
• Build a dataset of word-aligned speech for training or evaluating downstream audio models

// How WhisperX compares

WhisperX alongside other open-source audio, music & voice tools AI/TLDR tracks, ranked by GitHub stars.