# VibeVoice

> Open-source voice AI from Microsoft for long-form, multi-speaker TTS and ASR

- Category: Multimodal Generation / Audio, Music & Voice
- Language: Python
- Licence: MIT
- Repository: https://github.com/microsoft/VibeVoice
- Website: https://microsoft.github.io/VibeVoice/
- Canonical page: https://ai-tldr.dev/tools/vibevoice/

## In short

VibeVoice is a family of open-source voice AI models from Microsoft Research that covers both directions of speech: text-to-speech (TTS) and automatic speech recognition (ASR). It sits in the Audio, Music & Voice category of the Multimodal Generation tools and is open-source, written in Python and licensed MIT. To get started: install Transformers.

## Overview

VibeVoice is a family of open-source voice AI models from Microsoft Research that covers both directions of speech: text-to-speech (TTS) and automatic speech recognition (ASR). It is built around continuous speech tokenizers running at a low 7.5 Hz frame rate and a next-token diffusion framework, where a language model handles textual context and dialogue flow while a diffusion head produces the acoustic detail.

The lineup includes VibeVoice-TTS for long-form, multi-speaker audio, a smaller VibeVoice-Realtime-0.5B model for streaming text-to-speech, and VibeVoice-ASR for long-form transcription. The ASR model is published as a Hugging Face Transformers release, so you can load it with the standard AutoProcessor and model classes.

It fits the speech and audio corner of the multimodal space and is aimed at researchers and developers who need to generate or transcribe long stretches of conversational audio. Note that Microsoft removed the original VibeVoice-TTS code from the repository after misuse and frames the project as research-focused.

## What it does

- VibeVoice-TTS synthesizes conversational or single-speaker speech up to 90 minutes in a single pass
- Supports up to 4 distinct speakers in one dialogue with natural turn-taking and consistent voices
- VibeVoice-ASR transcribes up to 60 minutes of audio in one pass within a 64K token window
- ASR output is structured by Who (speaker), When (timestamps), and What (content), combining transcription and diarization
- Customized hotwords let you supply names or technical terms to improve recognition accuracy
- ASR is natively multilingual across more than 50 languages; TTS covers English, Chinese, and others

## Getting started

The ASR model ships through Hugging Face Transformers, which is the most direct way to try VibeVoice today.

1. Install Transformers

   VibeVoice-ASR is included in recent Transformers releases, so install or upgrade to a supported version.

```bash
pip install "transformers>=5.3.0"
```

2. Load the ASR model

   Load the processor and model by their Hugging Face id. device_map="auto" places the model on available hardware.

```python
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
```

3. Transcribe audio

   Build a transcription request from an audio source, generate, and decode the parsed result with speaker segments.

```python
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]

transcription = processor.decode(generated_ids, return_format="parsed")[0]
for speaker_transcription in transcription:
    print(speaker_transcription)
```

4. Try the other models

   For streaming text-to-speech, the VibeVoice-Realtime-0.5B model has a Colab notebook linked from the repository, and the TTS and ASR docs cover the full lineup.

## When to use it

- Generate long podcast-style audio with several speakers from a written script
- Produce single-speaker narration or audiobook segments that stay consistent over long passages
- Transcribe hour-long meetings or interviews with speaker labels and timestamps in one pass
- Add domain-specific names and terms as hotwords to improve transcription of technical content