# WhisperX

> Batched Whisper transcription with word-level timestamps and speaker labels

- Category: Multimodal Generation / Audio, Music & Voice
- Language: Python
- Licence: BSD-2-Clause
- Repository: https://github.com/m-bain/whisperX
- Canonical page: https://ai-tldr.dev/tools/whisperx/

## In short

WhisperX is a speech-recognition pipeline built around OpenAI's Whisper model. Plain Whisper produces accurate text but only gives timestamps at the utterance level, and those can be off by several seconds; it also has no native batching. It sits in the Audio, Music & Voice category of the Multimodal Generation tools and is open-source, written in Python and licensed BSD-2-Clause.

## Overview

WhisperX is a speech-recognition pipeline built around OpenAI's Whisper model. Plain Whisper produces accurate text but only gives timestamps at the utterance level, and those can be off by several seconds; it also has no native batching. WhisperX fixes both problems by wrapping Whisper in a four-stage pipeline — voice-activity detection, batched inference, forced phoneme alignment, and speaker diarization.

The alignment stage is what gives WhisperX its name recognition. After Whisper produces a transcript, a phoneme-based model in the wav2vec2 family is used to align the orthographic text back onto the audio, which yields a start and end time for every individual word rather than for each whole segment. A separate diarization stage, using pyannote-audio, partitions the audio by speaker and attaches a speaker label to each segment.

Because inference runs through the faster-whisper backend with VAD-based batching, the project reports roughly 70x realtime transcription using Whisper large-v2 while fitting in under 8 GB of GPU memory at beam size 5. WhisperX was accepted at INTERSPEECH 2023 and took first place in the Ego4d transcription challenge.

## What it does

- Batched inference on the faster-whisper backend — the project reports ~70x realtime with Whisper large-v2
- Word-level timestamps produced by forced phoneme alignment against wav2vec2 models
- Multi-speaker transcription: speaker diarization via pyannote-audio attaches speaker IDs to segments
- VAD preprocessing that trims silence, reduces hallucination, and enables batching without degrading WER
- Sentence-level transcript segments (via nltk sentence tokenization) for cleaner subtitles
- Command-line entry point plus subtitle output, including a --highlight_words flag that visualises word timings in the .srt

## Getting started

WhisperX ships on PyPI. GPU acceleration needs the CUDA toolkit installed first; CPU-only use can skip that step. Speaker diarization additionally requires a Hugging Face read token and acceptance of the pyannote model's user agreement.

1. Install from PyPI

   The simplest install is the published wheel. You can also run it without installing via uvx.

```bash
pip install whisperx
```

2. Transcribe an audio file

   Point the CLI at any audio file. With default parameters this uses the small Whisper model; add --highlight_words True to visualise word timings in the generated .srt.

```bash
whisperx path/to/audio.wav
```

3. Enable speaker diarization

   Generate a read token at huggingface.co/settings/tokens and accept the user agreement for the speaker-diarization-community-1 model, then pass the token with --hf_token to get speaker labels alongside the transcript.

```bash
whisperx path/to/audio.wav --hf_token YOUR_HF_READ_TOKEN
```

4. Install from source instead

   For development work, clone the repository and sync the environment with uv. The development version may contain experimental features; use the PyPI release in production.

```bash
git clone https://github.com/m-bain/whisperX.git
cd whisperX
uv sync --all-extras --dev
```

## When to use it

- Generate subtitle files where each word is timed precisely enough to karaoke-highlight during playback
- Transcribe multi-speaker recordings — interviews, panels, meetings — with a speaker label on every segment
- Batch-process a large archive of audio on a single GPU, where plain Whisper's lack of batching makes the job too slow
- Build a dataset of word-aligned speech for training or evaluating downstream audio models