What Is Speculative Decoding? Making LLMs Generate Faster

// In plain English

A large language model writes one token (roughly a word-piece) at a time. To produce each token, the whole multi-billion-parameter network runs once — a single forward pass. This step-by-step style is called autoregressive generation, and it's the main reason big models feel slow: a hundred output tokens means a hundred full passes through the model, one after another.

Speculative decoding is a trick to skip a lot of that waiting without changing the answer at all. The idea: keep a tiny, fast "draft" model alongside the big one. The draft model rapidly guesses the next few tokens. Then the big model checks all of those guesses in a single forward pass. Whatever it agrees with, it keeps for free; the first guess it disagrees with, it corrects. Repeat. On a good run you produce three, four, or five tokens for roughly the cost of one big-model step.

Here's the everyday version. A senior editor (the big model) writes carefully and slowly, weighing every word. An eager junior (the draft model) scribbles ahead: "The capital of France is Paris, a city...". The editor doesn't write from scratch — they read the junior's sentence in one glance, nod at the part that's right, cross out the first wrong word, and continue from there. The junior is cheap and quick; the editor is the one who guarantees quality. You get the editor's writing at closer to the junior's speed.

// Why it matters

Generating text with a large model is memory-bound, not compute-bound. For every token, the GPU has to read all of the model's weights out of memory to do the math — and that reading is the slow part (this is one reason LLMs need GPUs with very fast memory). The painful detail is that checking one token and checking several tokens in a single pass cost almost the same, because either way you've already paid to load the weights. The GPU spends most of a forward pass waiting on memory, with compute to spare.

Speculative decoding exploits exactly that spare compute. Instead of verifying one token per expensive pass, you verify a whole batch of proposed tokens in one pass — using capacity that was otherwise idle. If the cheap draft guesses well, you turn many slow sequential steps into far fewer, and latency drops sharply.

• Latency. Users feel the time-to-each-token. Cutting the number of big-model passes makes streaming responses visibly snappier — often roughly 2x faster, sometimes more, depending on the task.
• Cost and throughput. Fewer big-model passes per response means a server can handle more requests with the same hardware. For a high-volume API, that is real money.
• No quality tradeoff. Most speed tricks (smaller models, aggressive quantization, heavy caching) trade some accuracy for speed. Speculative decoding's headline feature is that, done correctly, it does not. The output distribution is provably unchanged.

This is why it now ships inside production serving engines (vLLM, TensorRT-LLM, llama.cpp and others) and behind many commercial APIs. It is one of the few "free lunch" optimizations in LLM inference — if you can find a draft model that agrees with the big one often enough.

// How it works

The loop has two players: a small draft model and the large target model (the one whose output you actually want). Each round does draft → verify → accept, and any rejected guess is repaired so the result stays exactly correct.

Step 1 — Draft: guess a few tokens ahead

Starting from the text so far, the cheap draft model runs its normal autoregressive loop for a short burst — say 4 tokens. Because it's small, those 4 quick passes are far cheaper than even one pass of the big model. The draft also records the probability it assigned to each token it picked; the verifier will need those numbers.

Step 2 — Verify: one big pass checks them all

Now the magic. You feed the original text plus all 4 drafted tokens into the target model as a single sequence, and run one forward pass. Thanks to the way attention works, that one pass produces the target model's own logits — its predicted next-token distribution — at every position at once. So in a single big-model step you learn what the target model would have said at each of the drafted spots, all in parallel. This parallel check is the whole reason the trick pays off.

Step 3 — Accept or correct (and stay exact)

Walk the drafted tokens left to right and compare each against the target model's distribution at that position. Each token is accepted with a probability set by how much the target model agreed with the draft. The moment one token is rejected, you stop, discard the rest of the draft, and sample one replacement token from a corrected distribution. This accept/reject rule ("speculative sampling") is carefully designed so the tokens you keep follow exactly the target model's own distribution — no bias is introduced.

Crucially, even in the worst case — the draft is rejected immediately — you still produce at least one valid token from that big-model pass (the corrected one). So you can never go slower in tokens-per-pass than plain decoding; speculative decoding only ever wins or breaks even on big-model passes. The diagram below traces a concrete round.

// Choosing a draft model

The whole method lives or dies on the acceptance rate — the fraction of drafted tokens the target model keeps. The best draft model is one that is both cheap and closely aligned with the target, because every rejected token is wasted draft work and a missed speedup.

Two forces pull against each other. A bigger draft model agrees with the target more often (higher acceptance) but costs more to run each guess. A tinier draft is nearly free but gets rejected more. The sweet spot is usually a small model from the same family as the target — for example a 1B-parameter draft for a 70B-parameter target — so they were trained similarly and tend to predict the same easy tokens.

Where do you get a matched draft? A few common patterns: pick a smaller official model in the same family; distill one specifically to imitate the target; or skip the separate model entirely and have the target model predict several tokens at once via extra lightweight "heads" — the approach popularized by Medusa and refined by EAGLE. Self-drafting is attractive because you serve only one set of weights.

// Where it fits among decoding ideas

Beginners often lump several different "decoding" topics together. They live at different layers, and speculative decoding sits on top of the others rather than replacing them.

Read it left to right. Greedy vs sampling is about which token you pick at one position. Autoregressive generation is the order — one token at a time, each feeding the next. Speculative decoding accelerates that loop while preserving whatever sampling setting you chose: if you run the target at temperature 0.7, the accepted tokens still follow the temperature-0.7 distribution. It is orthogonal to those choices, which is exactly why it can be a drop-in speedup.

It is also distinct from speedups that do change the math — like quantization or distilling a smaller model to serve directly. Those trade some accuracy for speed. Speculative decoding's defining property is that the target model still has the final say on every token, so quality is identical.

// Going deeper

Once the basic loop clicks, the interesting questions are about squeezing out more acceptance and handling real serving conditions. A few directions worth knowing.

How many tokens to draft? The draft length is a tunable knob. Draft too few and you under-use the parallel verify pass; draft too many and you waste effort on guesses that get rejected past the first mismatch. Some systems make this adaptive, drafting more when recent acceptance has been high and backing off when it's low.

Tree-based / multi-candidate drafting. Instead of one linear guess, draft a small tree of alternative continuations and verify many candidate paths in the same pass (the idea behind Medusa and EAGLE). More of the verify pass's spare compute gets used, so acceptance — and thus speedup — rises. The cost is more complex bookkeeping in the verifier.

Lookup / n-gram drafting. For repetitive output (long code blocks, copying text from the prompt), you can skip the draft model entirely and propose tokens by simple string lookup from context. It's almost free and accepts extremely well on copy-heavy text — a cheap complement to model-based drafting.

Batching tension. Speculative decoding shines most when the GPU is underused — small batches, low traffic, latency-sensitive single requests. When a server is already running large batches, the spare compute that the trick relies on is gone, so the win shrinks. Production engines weigh this and sometimes switch speculation off under heavy load.

Exactness, restated. The reason teams trust this in production is the proof behind speculative sampling: the accept/reject rule reproduces the target model's exact output distribution, token for token. So you debate speed and complexity, never correctness. The honest open problems are practical — finding or training a well-matched draft, choosing draft length, and keeping acceptance high across varied real traffic. If you want the foundations underneath it, revisit next-token prediction and how LLMs work; the speedup only makes sense once those are second nature.