Moshi

// Overview

Moshi is an open-source speech-text foundation model from Kyutai for real-time spoken dialogue. It is full-duplex, meaning it can listen and speak at the same time instead of waiting for you to finish a sentence. Moshi models two audio streams at once (its own voice and the user's) and also predicts the text of its own speech, an "inner monologue" that improves the quality of what it says.

It is aimed at researchers and developers who want to build or study low-latency voice assistants and conversational agents. Under the hood it pairs a 7B-parameter temporal transformer with a smaller depth transformer, and uses Mimi, a streaming neural audio codec that runs at a 12.5 Hz frame rate. The reported theoretical latency is around 160ms, with practical latency as low as 200ms on an L4 GPU.

In the speech and audio space, Moshi sits alongside text-to-speech and speech-to-text tools but goes further by handling a live two-way conversation in a single model. The repo ships three inference stacks: PyTorch for research, MLX for on-device use on Mac and iPhone, and Rust for production deployments.

// What it does

• Full-duplex dialogue: listens and speaks at the same time for natural turn-taking
• Mimi streaming codec compresses 24 kHz audio to a 12.5 Hz, 1.1 kbps representation with 80ms latency
• Inner-monologue text prediction alongside audio for higher-quality generation
• Three inference backends in one repo: PyTorch, MLX (Mac/iPhone), and Rust (production)
• Pretrained voices released on Hugging Face: Moshiko (male) and Moshika (female)
• Low practical latency, as low as 200ms on an L4 GPU

// Getting started

The PyTorch backend is the simplest way to try Moshi locally. You need at least Python 3.10 (3.12 recommended) and a suitable GPU; model weights download automatically from Hugging Face on first run.

pip install -U moshi

python -m moshi.server --hf-repo kyutai/moshika-pytorch-bf16

python -m moshi.client

pip install -U moshi_mlx
python -m moshi_mlx.local_web

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Building a real-time, full-duplex voice assistant that responds without waiting for a full pause
• Researching streaming speech-text models, audio codecs, and conversational latency
• Running an on-device voice agent on a Mac or iPhone via the MLX backend
• Deploying a production voice service using the Rust backend

// How Moshi compares

Moshi alongside other open-source audio, music & voice tools AI/TLDR tracks, ranked by GitHub stars.