SenseVoice

// Overview

SenseVoice is a speech foundation model with several speech-understanding heads rather than transcription alone: automatic speech recognition, spoken language identification, speech emotion recognition and audio event detection. The released SenseVoiceSmall checkpoint covers Mandarin, Cantonese, English, Japanese and Korean for ASR and language ID, and emits emotion and audio-event tags alongside the transcript — applause, laughter, crying, coughing, background music and similar interaction events.

The distinguishing design choice is a non-autoregressive end-to-end decoder. The repository's benchmark setup reports that at a comparable parameter count SenseVoiceSmall runs more than five times faster than Whisper-Small and fifteen times faster than Whisper-Large, which is what makes the "rich transcription" output — text plus emotion plus events in one pass — cheap enough to run over large archives. The README is careful about scope: the wider research reports training on over 400,000 hours and more than 50 languages, but the published checkpoint is the five-language one, and the benchmark comparisons are task- and language-specific.

In practice SenseVoice runs through FunASR, which supplies the surrounding pipeline: FSMN-VAD to split long recordings, CAM++ for speaker labels and ct-punc for punctuation when you compose a diarization pipeline. Speaker diarization is explicitly a composed pipeline, not an output of the SenseVoiceSmall checkpoint. The project also ships fine-tuning scripts, a service-deployment path that handles concurrent requests with clients in Python, C++, Java, C# and HTML, and prebuilt llama.cpp/GGUF runtime binaries for desktop CPU and GPU backends. Weights are published on ModelScope as `iic/SenseVoiceSmall` and on Hugging Face as `FunAudioLLM/SenseVoiceSmall`; the paper is arXiv:2407.04051.

// What it does

• One pass yields transcript, detected language, an emotion tag and audio-event tags — the project's "rich transcription" output
• SenseVoiceSmall covers Mandarin, Cantonese, English, Japanese and Korean for ASR and language ID
• Non-autoregressive end-to-end decoder — the README's benchmark reports >5x Whisper-Small and 15x Whisper-Large throughput at similar parameter count
• Bounded-memory long-audio script (`long_audio_no_vad.py`) for hour-scale recordings when VAD segmentation is not acceptable
• Composable diarization through FunASR with FSMN-VAD, CAM++ and ct-punc when speaker labels are needed
• Fine-tuning scripts plus a service-deployment pipeline with multi-concurrency and Python, C++, Java, C# and HTML clients
• Prebuilt llama.cpp / GGUF runtime archives for Linux, macOS and Windows, including Vulkan and Windows CUDA variants

// Getting started

SenseVoice is driven through FunASR. Install the dependencies, then load the SenseVoiceSmall checkpoint with a VAD model in front of it so long files are segmented before they reach the encoder.

pip install -r requirements.txt
pip install -U "funasr>=1.3.26"

from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model_dir = "iic/SenseVoiceSmall"

model = AutoModel(
    model=model_dir,
    trust_remote_code=True,
    remote_code="./model.py",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    device="cuda:0",
)

res = model.generate(
    input=f"{model.model_path}/example/en.mp3",
    cache={},
    language="auto",  # "zh", "en", "yue", "ja", "ko", "nospeech"
    use_itn=True,
    batch_size_s=60,
    merge_vad=True,
    merge_length_s=15,
)
print(rich_transcription_postprocess(res[0]["text"]))

python long_audio_no_vad.py meeting.mp3 \
  --output meeting.txt \
  --window-seconds 30 \
  --overlap-seconds 2

model = AutoModel(
    model="iic/SenseVoiceSmall",
    trust_remote_code=True,
    remote_code="./model.py",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    spk_model="cam++",
    punc_model="ct-punc",
    device="cuda:0",
)

pip install -U huggingface_hub
bash download-funasr-model.sh sensevoice
llama-funasr-sensevoice

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Transcribe Mandarin, Cantonese, English, Japanese or Korean audio where per-file language detection has to happen automatically
• Tag a call or support-recording archive with emotion labels alongside the transcript, in a single pass
• Detect non-speech interaction events — laughter, applause, coughing, background music — in media or meeting recordings
• Deploy self-hosted speech recognition with no Python runtime by shipping the prebuilt llama.cpp/GGUF binary

// How SenseVoice compares

SenseVoice alongside other open-source audio, music & voice tools AI/TLDR tracks, ranked by GitHub stars.