What Is Speculative Decoding? Faster Local Inference with a Draft Model

// In plain English

A language model generates text one token at a time. To produce each token, it runs a full forward pass over the whole network — billions of parameters, all of memory read once — and then appends a single word-piece. Then it does the entire thing again for the next token. For a local model on your own GPU, this is the slow part: most of the time isn't spent computing, it's spent shuttling those giant weights from memory for every single token.

Speculative decoding speeds this up with a clever trick: it pairs your big, accurate model with a tiny, fast draft model. The small model races ahead and guesses the next several tokens. The big model then checks all of those guesses in a single pass instead of one pass per token. Whatever guesses the big model agrees with are accepted for free; the first wrong guess is corrected and the rest are thrown away. Same output, fewer slow passes.

Picture an editor and a fast typist working together. The typist (the draft model) bangs out the next sentence as a quick guess. The editor (the big model) reads the whole sentence at once and says "yes, yes, yes — no, that word's wrong." Everything up to the mistake is kept; the editor fixes that one word, and the typist starts again from there. The editor still approves every word, so the final text is exactly what the editor would have written alone — it just got there with far less re-reading.

// Why it matters

Generating tokens one at a time is memory-bound, not compute-bound. On a consumer GPU running a 30B model, the bottleneck is reading every weight from VRAM for each token — the arithmetic units sit mostly idle, waiting on memory. That means a forward pass that checks five tokens at once costs almost the same wall-clock time as a pass that produces one. Speculative decoding exploits exactly this gap: if the draft's guesses are good, you get several accepted tokens for roughly the price of one.

• Lower latency. Interactive chat feels sluggish when tokens trickle out. Speculative decoding can lift tokens-per-second by 1.5×–3× on suitable workloads, with no change to the answer.
• No quality cost. Unlike quantization, which shrinks the model and can subtly degrade it, speculation leaves the big model's output identical. You don't have to choose between speed and accuracy.
• Better hardware use. A local GPU is often starved for work between memory reads. Verifying a batch of draft tokens fills that idle compute, so you extract more from the card you already own.
• Free to try. Both llama.cpp and vLLM support it with a couple of flags. If it doesn't help your workload, you turn it off — there's no risk to output quality.

Who cares most? Anyone running a large model locally for single-stream interactive use — coding assistants, local chatbots, agent loops where you wait on each response. Speculation shines exactly where you have spare compute and a latency problem, which is the typical situation for one user driving one big model on one GPU.

// How it works

The loop is draft, then verify. A small draft model proposes a short run of candidate tokens. The big target model evaluates all of them in one forward pass and decides — token by token — how many to keep. Then the cycle repeats from wherever the keep-streak ended.

Step 1 — the draft model guesses ahead

Starting from the text so far, the small draft model autoregressively generates a handful of tokens — say 4 or 8. Because it's tiny (often 10×–50× smaller than the target), these guesses are cheap and fast. They are just a proposal; nothing is committed yet.

Step 2 — the target model verifies in one pass

Now the big model processes the original text plus all the drafted tokens together in a single forward pass. Because transformers compute the probability of every position in parallel during a forward pass, the target gets its own prediction for each drafted slot at once — that's why checking 8 tokens costs about the same as generating 1.

Step 3 — accept the matching prefix, correct the first miss

Walking left to right, the target accepts each drafted token as long as it agrees with what the target itself would have produced (under a precise probabilistic acceptance test). At the first disagreement, it discards the draft from that point on, substitutes its own correct token, and the loop restarts. The math guarantees the kept sequence is drawn from exactly the target's own distribution — hence lossless.

A worked picture: the draft proposes the cat sat on the. The target verifies and agrees with the cat sat on but would have written a instead of the at the last slot. So it accepts the first four tokens, replaces the fifth with a, and starts the next round from there. Five tokens of progress, one big-model pass — when the draft is right, you fly; when it's wrong, you fall back to roughly normal speed for that round.

// When it helps and when it doesn't

Speculative decoding is not a universal speed button. Its payoff depends on two things: how predictable the text is (does the draft guess right often?) and how much idle compute you have to spend on verification. Match the workload to those conditions.

The tension with batching is worth understanding. Techniques like continuous batching win by keeping the GPU busy across many concurrent requests, which already removes the idle compute speculation needs. So speculation is most valuable for latency-sensitive, single-stream local use, and less so for a server packing dozens of users onto one card. They're not enemies — they optimize different regimes.

// Enabling it locally

The golden rule for picking a draft model: it must be small, fast, and from the same family as the target, so they share a tokenizer and a similar style of prediction. Common pairings are a 1B draft with a 30B–70B target. A draft that's too big erases the speedup; one from a different family (different tokenizer) won't verify correctly at all.

llama.cpp

llama.cpp exposes speculative decoding through a draft-model flag. You load your main GGUF as usual and point --model-draft at a small companion model from the same family; the related flags control how many tokens to draft per round.

# Main 70B target + small 1B draft from the same family.
# --draft-max sets how many tokens the draft proposes per round.
llama-server \
  --model       models/llama-70b-instruct.Q4_K_M.gguf \
  --model-draft models/llama-1b-instruct.Q4_K_M.gguf \
  --draft-max 8 \
  --draft-min 1 \
  -ngl 99

vLLM

vLLM enables speculation through a configuration object describing the draft method and how many tokens to speculate. The classic mode uses a separate small draft model; vLLM also supports draft-free variants (see Going deeper). Always confirm the exact key names against the version of vLLM you have installed — the speculative-decoding config has evolved across releases.

from vllm import LLM, SamplingParams

# A small draft model verified by the larger target model.
# num_speculative_tokens = how many tokens to draft each round.
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 5,
    },
)

out = llm.generate(
    ["Write a Python function that reverses a string."],
    SamplingParams(max_tokens=200),
)
print(out[0].outputs[0].text)

// Going deeper

The version above — a separate small model as the drafter — is the original form, sometimes called draft-model speculative decoding. A lot of the field since then has been about getting the speedup without maintaining a second model, plus tuning the knobs that govern acceptance.

Self-speculation and draft-free methods. Keeping a matched draft model around is a hassle, so newer methods generate the guesses from the target itself. Medusa bolts extra prediction heads onto the big model to propose multiple future tokens at once. EAGLE drafts from the target's own internal features for high acceptance. Lookahead decoding and prompt lookahead skip a model entirely, guessing tokens from n-gram patterns already in the text — cheap and surprisingly effective on repetitive or code-heavy output.

Tree-based verification. Instead of drafting one straight line of tokens, advanced methods draft a small tree of alternative continuations and verify many branches in a single target pass. This raises the odds that some path is accepted, squeezing out more tokens per verification — the approach behind several of the highest-throughput speculative methods.

Tuning the draft length. The tokens-per-round setting (--draft-max, num_speculative_tokens) is a balance. Too few and you rarely benefit; too many and rejected guesses waste work, since everything after the first miss is thrown away. The sweet spot depends on your acceptance rate, so it's worth sweeping a few values. Some implementations adapt the length automatically based on recent acceptance.

The honest limits. Speculation costs extra VRAM for the draft model and extra compute per round, so on a GPU that's already compute-bound (large batches, heavy serving workloads) the gains shrink or vanish. It also can't fix a fundamentally slow target model — it only removes redundant memory passes. The durable mental model: speculative decoding converts spare compute into fewer slow steps, lossless and reversible. Where you have idle compute and a latency problem, turn it on; where the GPU is already full, reach for batching or quantization instead. For the bigger picture of running fast locally, compare Ollama vs vLLM and check your hardware requirements.