Cohere · 2026-07-10 · notable
Cohere DSD — hardware-aware speculative decoding lands in vLLM
Cohere's Dynamic Speculative Decoding picks the draft-token count based on GPU pressure, so speculation still pays off at large batch sizes. Command A ran ~23% faster than fixed-K speculation at batches of 128 and 256.
Cohere's DSD picks the draft-token count from live GPU state, so speculation still speeds up inference at large batch sizes.
What is it?
Dynamic Speculative Decoding (DSD) is a new inference technique from Cohere Labs. DSD watches GPU compute and memory pressure at each batch step and adapts the number of draft tokens (K) on the fly, replacing the fixed-K schedule that most vLLM deployments still use. A vLLM pull request from the same author lands DSD upstream.
How does it work?
Standard speculative decoding drafts a fixed number of tokens per step. That helps at batch 1 but slows down at batch 128, where the extra drafts steal compute the main model needs. DSD models each GPU's throughput and picks the K that maximizes tokens per second per request, dropping to smaller K under load and rising when the model is bottlenecked on latency. On MoE models the optimal K is non-monotonic — DSD tracks that too.
Why does it matter?
Serving systems have been giving up speculative-decoding gains at production batch sizes because the fixed-K math stops working. DSD keeps the technique useful at batches of 128 and 256, where Command A ran roughly 23% faster than fixed-K speculation and 7.5% faster than vanilla decoding. The vLLM PR means anyone running vLLM can get the same speedup without patching the serving loop.
Who is it for?
inference-platform engineers running vLLM at production batch sizes
Try it
github.com/vllm-project/vllm/pull/32374