Throughput vs Latency: Tuning a Self-Hosted LLM Server

// In plain English

When you run your own model on an inference server, two numbers fight each other. Latency is how fast one person gets an answer. Throughput is how many people the server can serve in total per second. Push one up and you usually push the other down.

Picture a coffee shop with one barista. If the barista makes each drink start-to-finish before touching the next order, every customer gets their coffee fast — low latency — but the line out the door barely moves, so the shop serves few people per hour. Now imagine the barista pulls four espresso shots at once and steams milk for several orders together. Each individual customer waits a little longer, but the shop pushes far more cups out the door every hour — high throughput.

An LLM server works the same way. Processing requests one at a time gives each user the snappiest possible reply but wastes most of the GPU. Batching many requests together keeps the expensive GPU busy and serves more users, but each user waits slightly longer. Tuning a server is the art of choosing where on that line to sit.

// Why it matters

If you call a hosted API, the provider already tuned this for you. The moment you self-host — on a rented GPU, an on-prem box, or your own cloud — the trade-off becomes your problem, and getting it wrong is expensive in two different ways.

• Too slow for users. If you optimize purely for throughput by cramming huge batches together, individual replies crawl. A chat UI that takes six seconds to start typing feels broken, and users leave.
• Too underused to be affordable. A GPU that costs the same per hour whether it is 5% busy or 95% busy. If you run one request at a time to keep latency low, you may be paying for a $30k card to do $2k of work. Your cost per answer balloons.
• Wrong capacity planning. Without understanding the curve, you can't answer the basic business question: "how many GPUs do I need to serve N users at an acceptable speed?" You either over-provision and burn money or under-provision and time out under load.

Two very different people care about the two metrics. The user feels latency — they only experience their own request, and they judge the product on how fast their answer appears. Finance cares about throughput — they see the monthly GPU bill divided by the number of answers served. A good operator keeps both happy at once: fast enough for the user, busy enough for the accountant.

This is exactly why modern servers like vLLM exist. Their headline feature, continuous batching, is essentially a smarter answer to this trade-off than the naive choices above.

// How it works

To tune the trade-off you first have to measure it correctly. "Latency" for an LLM is not one number — it has two parts that users feel very differently.

The two latencies a user actually feels

• Time to first token (TTFT). The wait from sending the request to seeing the first word appear. This covers queueing plus the prefill step, where the model reads your whole prompt at once. Long prompts and a busy queue both make TTFT worse. This is the "is it even working?" feeling.
• Inter-token latency (ITL), also called time per output token. The gap between each streamed word after the first. This is the decode step — the model generates one token at a time. If you read at ~5 words/second, you barely notice an ITL under ~50ms; above that, the text visibly stutters.
• End-to-end latency. The total time for the full answer: TTFT + (ITL × number of output tokens). A short reply is dominated by TTFT; a long essay is dominated by ITL.

Against those sits the finance metric: throughput, usually measured in output tokens per second across all concurrent requests (sometimes also requests per second). A server might give one user 40 tokens/sec while delivering 2,000 tokens/sec in total across 60 users.

Why batching trades latency for throughput

A GPU is wildly parallel — it can multiply huge matrices in roughly the time it takes to multiply small ones. Running one request barely uses it. Batching means processing several requests' tokens together in one pass, so the GPU does far more useful work per step. That is why throughput climbs with batch size. The cost: each request now shares the GPU and waits its turn inside the batch, so per-token latency rises. The diagram below shows the whole path a request travels and where each metric is born.

Older servers used static batching: gather a fixed group of requests, run them all to completion, then start the next group. The problem is that a request asking for 20 tokens finishes long before one asking for 500, but its slot stays blocked until the longest request in the batch is done — wasting GPU and adding latency.

Continuous batching fixes this. The scheduler works at the token level: the instant any request finishes, a waiting request slots into its place mid-batch. The GPU never sits idle waiting for the slowest member, which is how vLLM and similar servers get high throughput and tolerable latency at the same time. Read the full mechanism in continuous batching.

// The knobs you actually turn

Tuning comes down to a small set of settings. Each one moves you along the latency-vs-throughput curve. Here is what each does and which way it pushes.

A subtle but important detail: KV-cache memory is the real ceiling on batch size. Every in-flight request stores its attention state (its KV cache) in GPU RAM, and that grows with the prompt and output length. Run out of KV-cache room and the server can't add more requests to the batch no matter what you set the batch size to — so freeing memory via quantization or a smaller model often raises throughput more than tweaking batch size directly. See local LLM hardware requirements for how that memory adds up.

// A method to find your setting

Don't guess. The reliable way to tune is to fix a latency target from the user's side, then push concurrency up until you hit it — the last setting that still meets the target is your answer. The shape you are looking for is universal:

Step by step

1. Write down a target users will accept, e.g. "p95 TTFT under 1s and ITL under 50ms." The p95 (95th-percentile) matters more than the average — the slow tail is what users complain about.
2. Use realistic traffic. Benchmark with prompt and output lengths that match your real workload. Tuning on 50-token prompts then serving 4,000-token prompts gives a useless answer.
3. Sweep concurrency. Run a load test (vLLM ships a benchmark_serving script; tools like k6, locust, or vegeta also work) at 1, 2, 4, 8, 16, 32… concurrent requests. At each level record TTFT, ITL, and total tokens/sec.
4. Plot throughput vs latency. Throughput rises with concurrency, then flattens once the GPU is saturated. Latency stays flat, then shoots up once requests start queueing.
5. Pick the last point under target. Set your concurrency limit there. You now serve the most users you can without breaking your latency promise.

# vLLM's own serving benchmark, run at several concurrency levels.
# Re-run with --max-concurrency 1, 4, 8, 16, 32 and compare the output.
for c in 1 4 8 16 32; do
  echo "=== concurrency $c ==="
  python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name random \
    --random-input-len 1024 \
    --random-output-len 256 \
    --max-concurrency $c \
    --num-prompts 200
done
# Read off: median/p99 TTFT, median ITL, and output token throughput.
# Your sweet spot = highest $c whose p95 TTFT and ITL still meet target.

// Common pitfalls

• Tuning on the wrong workload. Latency and throughput depend heavily on prompt and output length. A setting that's perfect for short chat turns can fall apart for long-document summarization. Tune for the traffic you actually serve.
• No concurrency cap. Letting unlimited requests in feels generous but punishes everyone — under a spike the queue grows without bound and TTFT becomes catastrophic. A finite cap plus graceful overflow is kinder than infinite acceptance.
• Confusing per-user and aggregate tokens/sec. "We get 2,000 tokens/sec!" can mean 2,000 for one user or 40 each for fifty users. Always state whether a throughput number is per-request or server-wide.
• Ignoring KV-cache limits. Cranking batch size past what GPU memory can hold causes preemption or out-of-memory errors, which hurt throughput. The memory ceiling is real and often the true bottleneck.
• Forgetting prefill vs decode are different. A workload dominated by huge prompts (lots of prefill) tunes differently from one generating long outputs (lots of decode). The bottleneck — and the right knob — shifts between them.

Finally, remember this trade-off is specific to serious self-hosting. If you are running a single model for yourself on a laptop with Ollama, throughput barely matters — you are the only user, so you tune purely for your latency. The throughput-vs-latency balancing act only becomes central once you serve many concurrent users from shared hardware, which is the gap between Ollama and vLLM.

// Going deeper

Once the basic sweep makes sense, a few advanced techniques let you bend the curve rather than just slide along it — buying more throughput at the same latency, or vice versa.

Chunked prefill. A giant prompt's prefill can monopolize a scheduler step and stall everyone else's decode, spiking ITL for active users. Modern servers split a long prefill into chunks and interleave them with ongoing decoding, smoothing latency without giving up throughput. It's often on by default; know it exists so you understand why a long prompt no longer freezes other users.

Prefix caching. When many requests share the same long prefix — a big system prompt, a shared document, a few-shot template — the server can compute that prefix's KV cache once and reuse it. That slashes prefill work, which improves both TTFT and throughput for free. If your traffic has a common prefix, enabling this is one of the biggest easy wins.

Speculative decoding. A small, fast "draft" model proposes several tokens, and the big model verifies them in one pass. When the draft is right, you get multiple tokens per expensive step, lowering ITL. It mainly helps latency at low-to-medium load; under heavy batching the GPU is already saturated, so the gains shrink.

Disaggregated serving. At large scale, some systems run prefill and decode on separate GPU pools, because the two phases stress the hardware differently (prefill is compute-bound, decode is memory-bandwidth-bound). Splitting them lets you tune and scale each independently — overkill for most teams, but it's where the frontier is heading.

The durable lesson: latency and throughput are two ends of one curve set by how the GPU is shared. You can't maximize both, but with continuous batching, the right concurrency cap, and a workload-matched benchmark you can find the point that keeps users happy and the GPU paid-for. Start from what an inference server is if any of the serving mechanics felt fuzzy, then return here to tune.