How to Reduce Latency in AI Voice Agents

// In plain English

A voice agent is a chain of steps that turns you talking into the computer talking back. You speak, a speech-to-text model writes down your words, a language model decides what to say, and a text-to-speech model turns that reply into sound. Latency is the silence in the middle — the gap between you finishing your sentence and the agent starting to speak.

Here is the key idea: in a real conversation, humans expect a reply in roughly a quarter of a second. Past about half a second of silence, the other person feels slow, hesitant, or broken. So a voice agent doesn't just need to be fast on average — it needs to start talking almost immediately, even if it hasn't finished thinking.

Think of a great simultaneous interpreter at a conference. They don't wait for the speaker to finish a whole paragraph and then translate it. They start speaking after the first few words and keep going, staying just behind the speaker the entire time. A slow voice agent is the opposite: it politely waits for every step to fully finish before the next one begins, and all that waiting stacks up into dead air.

// Why it matters

Text chat forgives slowness. A chatbot can think for two seconds and stream its answer word by word, and nobody minds — you're reading, the screen is filling, the wait feels productive. Voice has none of that cover. Silence on a phone line reads as something is wrong: the user repeats themselves, talks over the agent, or hangs up. A voice product that is correct but laggy still feels broken.

The trap is that each stage seems fast on its own, yet they add up. A transcription that settles in 300 ms, a model that takes 700 ms to start replying, and a voice that needs 400 ms to produce its first audio together leave the user sitting in 1.4 seconds of silence — long enough to feel awkward on every single turn. And that's before any network round-trips between the services.

• Conversation breaks down. Past ~500 ms of dead air, people start talking over the agent, which forces interruption handling and makes the whole exchange feel clumsy.
• Drop-off rises. On support and sales calls, every extra second of latency measurably increases the chance the caller gives up or asks for a human.
• Cost of the naive fix is huge. The instinct is "buy a faster model." But raw speed rarely closes the gap — the silence usually comes from waiting for whole stages to finish before the next one starts, which is an architecture problem, not a horsepower problem.

Who needs to care? Anyone building phone agents, voice assistants, live captioning with spoken responses, or in-car and wearable assistants. If a human is waiting to hear a reply rather than read one, latency is a top-three product metric, right next to accuracy and cost.

// How it works

To fix latency you first have to see it. The honest way is to build a latency budget: list every stage, measure how long each one takes, and find where the silence actually lives. A voice turn has three big stages, and each one can run in two modes — batch (wait for the whole input, then produce the whole output) or streaming (start producing output while input is still arriving).

The three stages of a voice turn

The single most important measurement in each stage is its time to first output — not how long the whole stage takes, but how long until it produces something the next stage can use. People shorten these to:

• End of speech detection — how fast you decide the user has stopped talking (voice activity detection, or VAD). End too late and you add pure dead air; end too early and you cut the user off.
• Time to first token (TTFT) — how long the LLM takes to emit its first word, not its last one. This is usually the biggest single chunk of the budget.
• Time to first audio byte (TTFB) — how long the TTS engine takes to produce the first playable audio, not the whole clip.

The naive pipeline waits at every step

The slow design treats each stage as a closed box: fully transcribe, then send the whole transcript to the LLM, then wait for the entire reply, then hand all of it to the TTS, then play. Every arrow in the diagram below is a place the user waits for a whole stage to finish. The latencies don't overlap — they stack.

The fix: stream every stage and chunk the TTS

Streaming means each stage starts emitting output before its input is complete, so the stages run at the same time instead of one after another. The LLM streams text token by token. As soon as you have a complete sentence of that text, you send it to the TTS and start playing the audio — while the LLM is still writing the rest of its answer. This sentence-by-sentence handoff is the single highest-leverage trick in voice latency, and it's why the streaming column above feels instant even though the total work is identical.

The chunking detail matters. You don't send one word at a time (the TTS would sound choppy and robotic) and you don't wait for the whole paragraph (that's back to batch). You buffer until you hit a natural boundary — a period, question mark, or a clear clause break — and flush that chunk to the synthesizer. The user hears sentence one while sentence two is being generated and sentence three is still a thought in the model's head.

// A worked latency budget

Numbers make this concrete. Below is the same turn measured two ways. The values are illustrative, but the shape is exactly what you see in practice: the naive version makes the user wait for the sum of every stage, while the streaming version makes them wait only for the first sentence to be spoken.

Notice the streaming column never adds the stages together. The user hears the first words after roughly the end-of-speech timeout plus the LLM's first token plus the TTS's first byte — and the rest of the reply is produced behind that audio, hidden by the fact that the user is already listening. The total compute didn't change; the ordering did.

Here's the core loop in pseudocode. The important line is the one that flushes a sentence to TTS the moment it's complete, rather than waiting for the stream to end.

buffer = ""

# llm_stream yields text tokens as the model writes them
for token in llm_stream(transcript):
    buffer += token

    # As soon as we have a complete sentence, speak it —
    # don't wait for the model to finish the whole reply.
    sentence, sep, rest = split_at_sentence_end(buffer)
    if sep:
        audio = tts_stream(sentence)   # also streaming: first byte ASAP
        play(audio)                    # user hears it NOW
        buffer = rest                  # keep the leftover for the next chunk

# Flush whatever's left after the stream ends
if buffer.strip():
    play(tts_stream(buffer))

// Common pitfalls

Most latency bugs aren't exotic. They're a handful of mistakes that quietly turn a streaming pipeline back into a batch one.

• Waiting for the full LLM response before speaking. This is the number-one mistake. If you call the model, wait for the complete reply, then synthesize it, you've thrown away the biggest win. Stream tokens and speak the first sentence early.
• Using a non-streaming TTS. If your synthesizer only returns a finished audio file, your time-to-first-byte equals time-to-whole-clip. Pick a TTS that streams audio as it generates, so playback can start on the first chunk.
• A lazy end-of-speech timeout. Many stacks wait a fixed, generous silence (say 800 ms) before deciding the user is done. That fixed wait is added to every turn. Tune VAD aggressively and use endpointing that reacts to natural pauses.
• Sequential service hops with cold starts. Each network round-trip and each cold-started model adds latency. Co-locate services, keep connections warm, and stream between them rather than POSTing a full payload and awaiting a full response.
• Chunking on the wrong boundary. Flushing the TTS on every word sounds choppy; flushing only on paragraphs reintroduces the wait. Split on sentence/clause boundaries — the sweet spot between smoothness and speed.
• Ignoring interruptions. If the user starts talking while the agent is mid-sentence, you must stop playback fast. Slow barge-in handling feels just as broken as slow first response.

// Going deeper

Once the basics — stream every stage, chunk the TTS on sentence boundaries, tune endpointing — are in place, the remaining gains come from a few more advanced ideas.

Speculative and filler responses. Some systems play a tiny, instant acknowledgement ("sure—", "let me check") generated locally while the real reply is still being produced. Used sparingly, this masks the unavoidable first-token wait the way a human says "mm-hmm" while gathering a thought. Overused, it sounds fake — so it's a tool, not a default.

Speech-to-speech and unified models. The classic stack has three separate models with three handoffs. Newer speech-to-speech models take audio in and emit audio out directly, collapsing STT, the LLM, and TTS into one streaming system. That removes the inter-stage waits entirely and can feel dramatically more natural — at the cost of less control over the intermediate transcript, which you often want for logging, moderation, or tool calls.

Smaller, closer models for the hot path. Latency is partly physics: audio has to travel to a server and back. Running a compact STT or even the first hop of the LLM on-device or at the network edge cuts round-trip time. A common pattern is a fast local model for the immediate response and a larger cloud model for harder turns.

Interruptions, turn-taking, and overlap. Real conversation is messy: people interrupt, pause mid-thought, and finish each other's sentences. Production voice agents spend real engineering on barge-in (stopping playback when the user speaks), endpointing (deciding when a turn truly ends), and handling overlapping speech. These aren't latency in the stopwatch sense, but they decide whether the conversation feels responsive.

The durable lesson: latency in a voice agent is rarely about one slow component. It's about ordering. The naive pipeline waits for each stage to finish before starting the next, so silences add up. The fast pipeline overlaps the stages by streaming, so the user hears the first words while the rest is still being made. Build a latency budget, measure perceived silence, stream everything, and chunk the speech — in that order. To round out the stack, see how text-to-speech and speech-to-text work under the hood.