whisper-timestamped

// Overview

Whisper transcribes well but times poorly: the models were trained to predict segment timestamps to roughly one-second accuracy and cannot produce word boundaries at all. whisper-timestamped, from LINAGORA's LinTO team, adds them. It applies dynamic time warping to the model's cross-attention weights — the approach demonstrated in Jong Wook Kim's Multilingual ASR notebook — and assigns a confidence score to every word and every segment along the way.

The implementation is deliberately a drop-in. It is an extension of the `openai-whisper` package and is meant to work with any version of it: import `whisper_timestamped` instead of `whisper`, call `transcribe(model, ...)` instead of `model.transcribe(...)`, and the output gains a `words` key on every segment with start, end and confidence. The CLI mirrors Whisper's, adding a CSV output format and writing word timestamps into the SRT, VTT and TSV files as well.

Two design choices matter in practice. When beam search is off, word alignment is computed on the fly as each speech segment is decoded, so no additional inference pass is needed — which is also why the defaults favour greedy decoding with no temperature fallback (`--accurate` restores Whisper's beam-search defaults). And memory use was treated as a first-class concern, so long files cost little more than a plain Whisper run.

The optional voice-activity detection step is the practical fix for a well-known Whisper failure: predicting phrases like "Thanks for watching!" over silence, a habit learned from the training data. Running silero (the default) or auditok before the model removes the non-speech regions that trigger it. The project's README is candid about the alternative approaches too — it explains why cross-attention DTW avoids the drawbacks of wav2vec-based alignment (one model per language, extra memory, awkward character normalisation, fragility around disfluencies) and why reading timestamp-token probabilities, as whisper.cpp and stable-ts do, can go badly out of sync. The maintainers flag the extension as experimental and note it can significantly affect performance.

// What it does

• Word-level start/end timestamps from DTW over the model's own cross-attention weights — no second neural network
• A confidence score on every word and every segment
• No extra inference pass when beam search is off: alignment happens on the fly as each segment is decoded
• More accurate segment start/end estimation than stock Whisper, whose timestamps round to about a second
• Optional VAD (silero, auditok, auditok:v3.1) before transcription, to stop hallucinated text over silence
• Drop-in API and CLI compatibility with openai-whisper, plus a CSV output format and word timestamps inside SRT, VTT and TSV
• Works with fine-tuned Whisper checkpoints from the Hugging Face Hub or a local folder
• A `--plot` option that renders the word alignment, and helpers such as `remove_non_speech` for use on their own

// Getting started

Install from PyPI on top of Python 3.9+ (3.7 is the floor) and ffmpeg. Optional extras: matplotlib for the alignment plot, onnxruntime and torchaudio for VAD, transformers for Hugging Face checkpoints.

pip3 install whisper-timestamped

import whisper_timestamped as whisper

audio = whisper.load_audio("AUDIO.wav")

model = whisper.load_model("tiny", device="cpu")

result = whisper.transcribe(model, audio, language="fr")

import json
print(json.dumps(result, indent = 2, ensure_ascii = False))

whisper_timestamped audio1.flac audio2.mp3 audio3.wav --model tiny --output_dir .

from whisper_timestamped import remove_non_speech

audio_speech, segments, convert_timestamps = remove_non_speech(audio, vad="silero")

whisper_timestamped --model NbAiLab/whisper-large-v2-nob <...>

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when subtitles need word-level timing — karaoke-style highlighting, word-synced captions, clip extraction on an exact word
• Reach for it when a transcript has to be searchable down to the word, with a confidence score to flag what a human should check
• Reach for it when Whisper keeps inventing text over silence and you need VAD in front of it
• Reach for it when whisperX's wav2vec alignment is awkward for your language set and you would rather not carry a second model per language

// How whisper-timestamped compares

whisper-timestamped alongside other open-source audio, music & voice tools AI/TLDR tracks, ranked by GitHub stars.