WhisperLiveKit

// Overview

WhisperLiveKit (WLK) is a self-hosted speech-to-text pipeline built for live audio rather than finished recordings. Running Whisper on short chunks of a stream loses context and cuts words mid-syllable, because the model was trained on complete utterances; WLK instead applies simultaneous-speech research — LocalAgreement from WhisperStreaming and the AlignAtt policy from Simul-Whisper/SimulStreaming — to buffer intelligently and commit text incrementally as the speaker talks.

The project ships as a server plus a bundled web UI. One backend serves multiple concurrent users, voice-activity detection keeps it idle when nobody is speaking, and each WebSocket session can set its own language, translation target and terminology hints. On top of transcription it does streaming speaker diarization via Sortformer, and simultaneous translation across 200 languages through the maintainer's NLLW work.

Backends are pluggable: Whisper family models, Mistral's Voxtral Mini, FunASR SenseVoiceSmall, NVIDIA Canary, and a causal streaming encoder for Qwen3-ASR. For clients, WLK exposes a native WebSocket stream, an OpenAI-compatible `/v1/audio/transcriptions` REST endpoint and a Deepgram-compatible WebSocket, so existing SDK code can point at a local server instead of a paid API. A SwiftUI macOS client and a Chrome extension for capturing page audio are included in the repository.

// What it does

• Streaming transcription using LocalAgreement and AlignAtt policies, so partial output is committed as the speaker talks instead of after each chunk
• Real-time speaker diarization with Streaming Sortformer, plus simultaneous translation to and from 200 languages
• One server, many concurrent sessions, with per-session language, translation target and terminology context set as WebSocket query parameters
• OpenAI-compatible REST endpoint and Deepgram-compatible WebSocket alongside the native `ws://.../asr` stream
• Swappable backends — Whisper, MLX Whisper on Apple Silicon, Voxtral, FunASR SenseVoiceSmall, Canary and Qwen3-ASR
• A `wlk` CLI that also transcribes files offline and generates subtitles without running a server

// Getting started

Install the package, start the server, and open the bundled web UI. Optional extras pull in the heavier backends — several conflict with one another by design and must live in separate environments; the authoritative list is `[tool.uv].conflicts` in the project's pyproject.toml.

pip install whisperlivekit

wlk --model base --language en

# or let it pull the model for you
wlk run whisper:tiny

wlk transcribe meeting.wav
wlk transcribe --format srt podcast.mp3 -o podcast.srt
wlk models
wlk bench

curl http://localhost:8000/v1/audio/transcriptions -F file=@audio.wav

# Python
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

uv sync --extra cu129 --extra diarization-sortformer

# Apple Silicon, Voxtral backend
pip install -e ".[voxtral-mlx]"
wlk --backend voxtral-mlx

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Add live captions to a meeting, stream or call without sending audio to a third-party transcription API
• Replace a paid Deepgram or OpenAI transcription endpoint in an existing app by changing only the base URL
• Transcribe multilingual conversations and translate them as they happen, with per-session language settings
• Produce diarized transcripts that attribute each line to a speaker in real time

// How WhisperLiveKit compares

WhisperLiveKit alongside other open-source audio, music & voice tools AI/TLDR tracks, ranked by GitHub stars.