What Are Speech-to-Speech Models? Voice In, Voice Out

// In plain English

A speech-to-speech model takes spoken audio in and gives spoken audio back, without turning your words into written text in between. You talk, it listens to the actual sound, and it answers out loud — laughter, pauses, a rising tone of urgency and all. The voice goes in one end and a voice comes out the other, with a single model in the middle that understands and speaks sound directly.

Compare that to how most voice assistants worked for years. They used a relay race of three runners: first speech-to-text writes down what you said, then a language model reads that transcript and decides what to reply, then text-to-speech reads the reply out loud. Three separate models, three handoffs. The moment your voice became plain text, everything about how you said it — the sarcasm, the worry, the half-finished sentence — was thrown away.

Here's an everyday analogy. The three-stage pipeline is like asking a question through a translator who only passes along a written summary: "the customer wants a refund." Accurate words, but the manager on the other end never hears that the customer is furious or about to cry. A speech-to-speech model removes the note-passing. It is the person who hears your voice directly and answers in real time, reacting to your tone, not just your words.

// Why it matters

If text already works, why bother feeding raw audio to a model? Because human speech carries a huge amount of meaning outside the words, and the old pipeline silently destroys most of it. Three problems drove the shift to speech-to-speech.

• Tone and emotion survive. "Sure, that's fine" can mean genuine agreement or seething frustration — and only the sound tells you which. This wrapping of pitch, rhythm, stress, and timing is called prosody. Flatten audio to text and prosody vanishes. An audio-native model keeps it, so it can both hear that you're upset and answer in a calm, matching voice.
• Latency drops. A relay race is only as fast as all three runners plus the handoffs. Each stage adds delay, and the text-to-speech step usually can't start talking until the language model has finished thinking. A single model can begin forming its spoken reply almost as you stop talking, which is what makes natural back-and-forth — and being interrupted mid-sentence — actually feel real.
• Non-word sounds matter. Laughter, sighs, a sharp intake of breath, two people talking over each other, a child crying in the background, a song hummed off-key. None of that is a word, so none of it reaches a text transcript. A speech-to-speech model can perceive and even produce these sounds because it never leaves the audio domain.

Who cares in practice? Anyone building a voice agent that should feel like a person: phone-support bots that need to sense an angry caller, real-time language tutors that correct your pronunciation (impossible from a transcript), accessibility tools, hands-free assistants, and interview or roleplay practice apps. If the way something is said is part of the job, the text middleman is throwing away your most important signal.

// How it works

To see what's different, put the two designs side by side. The classic pipeline passes text between three specialist models. The speech-to-speech model passes audio representations through a single model that was trained to handle sound from start to finish.

Turning sound into tokens

A language model can't read a raw waveform any more than it can read a JPEG. The key idea that made speech-to-speech possible is the audio codec (or neural audio tokenizer): a small neural network that chops continuous sound into a stream of discrete audio tokens — the sound equivalent of the text tokens an LLM already uses. Crucially, these tokens capture not just which sound but how it was voiced: pitch, energy, timing. There is a matching decoder that turns a stream of audio tokens back into a real waveform you can play.

Once audio is just tokens in a sequence, a transformer can treat it almost the way it treats text. The model is trained on enormous amounts of speech (often paired with text too) to predict the next audio token. Predict spoken tokens one after another and you have a model that talks. That's why these are sometimes called audio-native LLMs: the same next-token machinery you'd use for text, pointed at sound.

Why it can answer so fast

Because there's no waiting for a full transcript and a separate voice engine, the model can stream: it processes your incoming audio tokens as they arrive and starts emitting reply tokens almost immediately. That streaming, full-duplex design (listening and speaking can overlap) is what lets a good speech-to-speech system be interrupted, pause, and resume the way a real conversation does. Many real systems still keep a text channel running alongside the audio — the model can output a transcript at the same time — so you get the natural voice and a readable log.

// What the text middleman throws away

The clearest way to understand speech-to-speech is to list exactly what a text transcript cannot hold. Every item below is information the pipeline destroys the instant it writes your words down — and that an audio-native model can keep.

This is also why speech-to-speech and speaker diarization (figuring out who spoke) are natural companions: both live in the part of the audio that text formats simply don't have a slot for.

// The tradeoffs, and when to use which

Speech-to-speech is not strictly better — it trades control for naturalness. Because the model speaks directly from sound, you lose some of the clean checkpoints the pipeline gave you, and you gain new headaches around moderation and debugging.

• Less controllable output. With separate TTS you can edit the exact text before it's spoken, swap a polished voice, or run the reply through a profanity filter. An end-to-end model decides the words and the delivery in one shot, so steering precisely what it says is harder.
• Harder to log and audit. There's no guaranteed clean transcript in the middle, so you can't always inspect "what did the model decide to say" as plain text. Production systems work around this by making the model emit a text channel too, but it's an extra moving part, not a free byproduct.
• Moderation is trickier. Catching a banned phrase is easy on text and much harder on a stream of audio tokens. Safety filters that assume a transcript don't slot in cleanly.
• Fewer off-the-shelf options and higher cost. Audio tokens are far denser than text, so context is consumed quickly and these models can be more expensive to run than a tuned three-stage pipeline.
• Voice and language control is coarser. Getting a specific branded voice or guaranteeing a language can be less precise than picking a dedicated TTS voice — overlapping with voice cloning concerns.

A useful rule of thumb: reach for speech-to-speech when the experience is the product — natural, fast, emotionally aware conversation. Reach for the pipeline when you need tight control, cheap operation, strict moderation, or a clean text record. Many teams even start on the pipeline to ship fast and move conversational, latency-sensitive flows to speech-to-speech later.

// Going deeper

Once the basics click, a few subtler points separate a toy demo from a real system.

"End-to-end" is a spectrum, not a binary. Few production systems are purely audio in, audio out with nothing else. A common middle design is speech-to-speech with an interleaved text track: the model jointly produces audio and a transcript, giving you natural prosody plus a readable, moderatable log. Other systems keep a real LLM "brain" in the middle for reasoning and tool use, wrapped in audio encoders and decoders. When someone says "end-to-end," ask where exactly text still appears.

Full-duplex and turn-taking are their own hard problem. Knowing when the user has finished talking, when it's okay to start replying, and how to handle being interrupted is not solved by the audio model alone. Systems add voice activity detection and barge-in handling so the model stops talking the instant you cut in — the difference between a smooth conversation and an awkward walkie-talkie.

Tool use and grounding still need text — or structure. A voice agent that has to look up an order or call an API generally needs to produce a structured request, which is a text-shaped task. So real voice agents often combine audio-native conversation with text-based tool calling and retrieval under the hood. The audio model handles talking; a text-capable layer handles acting on the world.

Evaluation gets fuzzier. Text systems can be scored on transcription accuracy. Speech-to-speech adds questions text metrics can't capture: does the voice sound natural, is the emotion appropriate, is the latency low enough to feel live, does it handle interruptions gracefully? These often need human listening tests, not just an automatic score.

Safety and consent deepen. A model that speaks fluidly and can mimic emotion raises real misuse concerns — convincing scam calls, impersonation, and the voice-cloning consent questions that come with any system that can reproduce a voice. The same realism that makes speech-to-speech feel human is exactly what makes responsible deployment, watermarking, and disclosure matter. From here, the natural next steps are the building blocks it sits on: speech-to-text, text-to-speech, and the broader idea of multimodal AI that treats sound, text, and images as one shared language of tokens.