What Is a Realtime Voice API? Speech-to-Speech Agents Explained

// In plain English

A realtime voice API lets your app talk with a person the way two people talk on a phone call. You open one streaming connection, the user starts speaking, and audio flows both ways at once: their voice goes in as a continuous stream, the AI's spoken reply comes back as a continuous stream — and either side can interrupt the other mid-sentence.

Contrast that with the old way of building a voice bot. You'd record the whole question, send the file to a speech-to-text model, wait for a transcript, send that text to an LLM, wait for the written answer, send that to a text-to-speech model, wait for an audio file, then finally play it. Three round trips, each one a stop-and-wait. The user sits in silence for two or three awkward seconds before anything comes out.

Think of the difference between sending voice memos back and forth versus a live phone call. With voice memos, you speak, hit send, wait, get a reply, listen, then speak again — turn by turn, with a gap each time. A live call has no gaps: you hear the other person while they speak, you can jump in with "wait, no —", and they adjust instantly. A realtime voice API is the live-call version of an AI assistant.

// Why it matters

Voice feels natural only when it's fast. Human conversation has a rhythm: we expect a reply within a few hundred milliseconds of finishing a sentence. Cross roughly one second of silence and the interaction stops feeling like a conversation and starts feeling like a kiosk. The whole point of a realtime voice API is to live inside that human-comfortable latency budget, which the classic stitched-together pipeline almost never can.

Here are the concrete problems it solves for a builder:

• Latency. A traditional pipeline pays the full cost of each stage in series, and each stage waits for the previous one to finish before it starts. A realtime session overlaps the work — the model can begin understanding and even start responding before the user has stopped talking — so perceived delay drops from seconds to a few hundred milliseconds.
• Interruptions (barge-in). Real talk is messy. People cut in, change their mind, say "actually, never mind." A turn-based pipeline has no concept of this; it will keep playing a now-irrelevant answer over you. A realtime API detects that you've started speaking, stops its own audio, and listens — the single biggest thing that makes a voice agent feel alive instead of robotic.
• Tone and emotion. When you transcribe speech to plain text first, you throw away how it was said — the hesitation, the sarcasm, the rising panic. A unified speech-to-speech model can hear that nuance directly and answer with matching prosody (a calm voice for a frustrated caller), because the audio never gets flattened into text.
• Fewer moving parts. One API and one connection replace three separate services you had to wire, time, and keep in sync. Less glue code, fewer places for latency and bugs to hide.

Who cares? Anyone building phone-based customer support, in-car or hands-free assistants, live language tutors, drive-through ordering, accessibility tools, or any product where a person would rather speak than type. If the experience has to feel like talking to a person, the turn-by-turn pipeline's silences will sink it.

// How it works

A realtime voice API runs over a persistent, bidirectional connection — usually a WebSocket or a WebRTC session — instead of one-shot HTTP requests. "Bidirectional" is the key word: your app streams microphone audio up in tiny chunks (often 20–40 milliseconds each) while the server streams the AI's spoken reply down at the same time, on the same connection. Nobody waits for the other to be completely done.

Because audio is flowing continuously, the system needs to figure out when a turn ends on its own. That job is voice activity detection (VAD): the server listens for a pause and decides the user has finished their thought, which triggers the model to respond. Good VAD is what makes the back-and-forth feel snappy; bad VAD either cuts people off or leaves long dead air.

There are two ways the box labeled Model can be built, and the difference is the heart of this topic.

Approach 1 — the classic cascade (STT → LLM → TTS)

You chain three specialist models. Speech-to-text turns audio into a transcript, an LLM reads the transcript and writes a reply, and text-to-speech turns that reply back into audio. A realtime framework can still make this feel responsive by streaming between the stages — feeding partial transcripts to the LLM and starting TTS on the first sentence of the answer before the rest is written — but you're orchestrating three separate systems.

Approach 2 — the unified speech-to-speech model

A single model takes audio in and produces audio out directly, without a text transcript in the middle as a hard step. Because it never flattens speech to text, it keeps the tone, timing, and emotion of what was said, and it has fewer internal hops to add delay. This is the "speech-to-speech" or "omni" style of realtime model. (Most still expose a text transcript so you can log and read the conversation — they just don't depend on round-tripping through text to think.)

Either way, two features make the session feel human. Partial (streaming) transcripts mean you get words as they're recognized — useful for showing live captions or letting the model start thinking early — instead of one block of text at the end. Barge-in handling means when the user starts talking while the AI is speaking, the API tells you to stop playback immediately and switches to listening, so the user is never talking over a wall of robot audio.

// A typical session, step by step

Most realtime voice APIs follow the same event-based shape, whatever the vendor. You open a socket, configure the session once, then exchange small JSON or binary events until you hang up. The pattern looks roughly like this:

1. open WebSocket  ->  server: "session.ready"
2. you send:  session.update { voice, language, instructions }
3. loop:
     you  -> input_audio.chunk   (mic, 20-40ms of audio)
     server -> transcript.partial  "how do I res..."
     server -> speech_started      (VAD heard the user)
     ... user pauses ...
     server -> speech_stopped      (end of turn)
     server -> response.audio.chunk  (reply audio, streamed)
     server -> transcript.partial    (text of the reply)
     [if user talks now] you -> response.cancel  (barge-in)
4. close WebSocket

Notice three things. Audio goes both directions in the same loop, not in request/response turns. The server pushes events, not one final answer, so your client must react to a stream. And you are responsible for the barge-in cancel: when your mic detects the user speaking over the reply, you send a cancel and stop playing the buffered audio you already received.

// When to use which approach

Unified speech-to-speech isn't automatically the right call. The cascade trades some latency and nuance for control and flexibility, and that trade is often worth it. Use this rough guide:

A common middle path: use a unified realtime model for the conversational layer (so it feels alive) but give it tools it can call out to for facts and actions — looking up an order, checking a calendar, running a retrieval step. That keeps the snappy speech-to-speech feel while letting a more controllable system handle the parts that need to be exactly right.

// Common pitfalls

Realtime voice is genuinely harder to get right than text chat, because timing bugs you can't see in a transcript ruin the feel. The usual traps:

• Not handling barge-in. The single most common reason a demo feels robotic. If the AI keeps talking after the user interrupts, the illusion of conversation collapses. Cancel the response and flush your audio buffer the instant your mic detects speech.
• Ignoring network jitter. Audio is unforgiving — a 300ms hiccup is an audible glitch. Buffer a little on both ends, prefer WebRTC for unreliable networks (it's built for live media), and assume packets arrive late and out of order.
• Sloppy turn detection. Too-aggressive VAD cuts people off mid-thought; too-lax VAD leaves long silences that feel like the AI froze. Tune the end-of-turn sensitivity for your domain, and expect to handle people who pause to think.
• Echo and feedback. On a speakerphone, the AI hears itself and tries to respond to its own voice. You need echo cancellation, or the model will talk to itself in a loop.
• Forgetting it's still an LLM. The voice layer is new, but the brain behind it can still hallucinate, refuse, or get confused — and now it's doing so out loud, in real time, with no chance for the user to re-read. Ground it and constrain it as carefully as any other LLM feature.

// Going deeper

Once the basic talk-back loop works, the interesting problems are about making it robust, controllable, and observable. A few directions worth knowing.

Tool calling mid-conversation. The most capable voice agents don't just chat — they do things. While speaking, the model can pause to call a function (check an order, book a slot, query a database), then resume the conversation with the result, often with a natural filler like "let me check that for you." Wiring tools into a streaming session, and deciding what the agent says during the wait, is where realtime voice meets agent design.

Transcripts even in speech-to-speech. Even when no text step is required for the model to think, you almost always want a text transcript for logging, analytics, compliance, and debugging. Most realtime APIs emit one as a side channel. If your domain is regulated, treat that transcript as a first-class requirement, not an afterthought — and verify it actually matches the audio.

Speaker separation. A unified model usually assumes one user and one assistant. The moment you have several people in a room — a meeting, a family at a drive-through — you need speaker diarization to tell who said what, which most realtime voice pipelines don't do for you out of the box.

Transport choice matters more than it looks. WebSocket is simple and works great on stable connections; WebRTC is more complex but is purpose-built for live media on lossy networks (it handles jitter, packet loss, and echo cancellation natively), which is why most phone and mobile voice products lean on it. The right transport is often the difference between "flawless in the office" and "unusable on a train."

It's one slice of a bigger picture. Voice is one modality among several — the same systems increasingly take images and video too. Understanding realtime voice as a streaming, interruptible session is a great on-ramp to the broader world of multimodal AI, where text, audio, and vision share one model. The durable lesson: a realtime voice agent lives or dies on timing — latency, turn-taking, and barge-in — far more than on raw model quality.