Tokens per Second Explained: How to Measure Local LLM Speed

// In plain English

When you run a local LLM, the model doesn't write whole sentences at once. It produces text one token at a time — a token being a small chunk of text, roughly three-quarters of a word on average. Tokens per second (often written tok/s or t/s) is simply how many of those chunks the model can produce each second. It's the headline number people quote when they say a setup feels "fast" or "painfully slow."

Picture someone reading a book aloud to you. A fast, fluent reader runs ahead of your eyes; a slow one makes you wait, word by word, for the sentence to finish. Tokens per second is the model's reading-aloud speed. At 40 tok/s the words pour out faster than you can read them. At 3 tok/s you watch each word appear like a 1990s dial-up web page loading line by line.

There's a catch most people miss: there isn't one speed. A model has two very different jobs in a single request, and each has its own tokens-per-second number. First it has to read your prompt — every word of the question, the system instructions, and any pasted documents. Then it has to write the answer. These two phases run at wildly different speeds and are bottlenecked by completely different things. If you only ever look at one number, you'll be surprised when a long prompt suddenly feels slow even though the model "benchmarks fast."

// Why it matters

If you're choosing a model or a piece of hardware to run it on, tokens per second is the single number that decides whether the experience is pleasant or unusable. A model can be smart, well-quantized, and free, and still be the wrong choice because it crawls on your machine. Speed is a feature.

It matters most because your comfortable speed depends entirely on what you're doing:

• Interactive chat. You're reading along as the model types. The honest target is to beat your own reading speed — most people read 5 to 8 words per second, so anything above roughly 10 tok/s feels live, and 20+ feels effortless. Below ~5 tok/s, chatting becomes a chore.
• Coding assistants and autocomplete. Here latency is brutal. You want the suggestion now, before you've typed the next character. Slow generation breaks your flow, so coding setups push for high tok/s and short prompts.
• Bulk or batch jobs. Summarizing 10,000 documents overnight? You don't watch it. Total throughput matters, not per-token feel. A setup that's too slow to chat with can be perfectly fine for a job that runs while you sleep.

It also matters for budgeting hardware. The reason a bigger or differently-quantized model runs slower is usually memory bandwidth, not raw compute — and that single fact, explained below, predicts performance better than the GPU's advertised TFLOPS. Understanding tok/s is how you read a benchmark and know whether a given GPU, Mac, or laptop will actually be comfortable before you spend the money. For the hardware side, see local LLM hardware requirements.

// How it works

A single request flows through the two phases, and each produces its own tokens-per-second number. Understanding what bottlenecks each one is the whole game.

Prefill: reading the prompt

In prefill, the model processes every token of your prompt at once. Because all the tokens are already known, the hardware can crunch them in parallel — a big matrix multiply that uses the GPU's compute units efficiently. Prefill is therefore compute-bound and usually fast: hundreds or even thousands of tokens per second. The cost of prefill scales with prompt length, so a giant prompt (a long document, a big system message) takes real time before any answer appears. That delay is time to first token (TTFT) — the pause you stare at after hitting enter.

Decode: writing the answer

Decode is different and slower. The model generates one token, then must feed that token back in to generate the next — it can't predict token 50 until it has produced token 49. This is sequential, so there's no parallelism to exploit. For each and every token, the hardware has to read the model's entire set of weights out of memory. That's the bottleneck: not how fast the chip can multiply, but how fast it can move the weights from memory to the compute units. Decode is memory-bandwidth-bound.

This is why the headline spec to check before buying isn't the GPU's TFLOPS — it's its memory bandwidth (GB/s). It's also why an Apple Silicon Mac with unified memory can run big models respectably: lots of fast, wide memory. And it's why a model that barely fits in your VRAM runs fine but the moment it spills into slow system RAM, decode falls off a cliff — system RAM has a fraction of the bandwidth of GPU memory.

// Reading the timing output

You don't have to guess these numbers — llama.cpp prints them after every generation, and Ollama exposes them too. Once you know which line is which, you can diagnose a slow setup in seconds. (For background on the engines, see what is llama.cpp and what is Ollama.)

Here's a typical llama.cpp footer, lightly trimmed:

prompt eval time =    512.30 ms /   256 tokens (   2.00 ms per token,  499.71 tokens per second)
       eval time =   4810.55 ms /   200 tokens (  24.05 ms per token,   41.58 tokens per second)
      total time =   5322.85 ms /   456 tokens

• prompt eval = prefill. Here, 256 prompt tokens read at ~500 tok/s — fast, because it's parallel and compute-bound.
• eval = decode. Here, 200 tokens generated at ~42 tok/s — this is the speed you feel as the answer streams. This is the number to quote.
• ms per token is just the reciprocal. 24 ms/token = 1000 ÷ 24 ≈ 42 tok/s. Lower ms is better.

Ollama gives you the same breakdown if you ask for it. Run with verbose mode and read prompt eval rate (prefill) and eval rate (decode):

ollama run llama3.2 --verbose
# after a reply, Ollama prints:
#   prompt eval rate:   480.12 tokens/s   <- prefill
#   eval rate:           38.94 tokens/s   <- decode (the one to watch)

// What counts as 'fast enough'?

There's no universal pass mark — it depends on the task. The table below is a practical feel-guide for decode speed (the generation number). Treat it as rough orientation, not a benchmark.

Two honest caveats. First, prefill speed matters separately for long prompts: a 30 tok/s decode setup still feels slow to start if you paste a 10,000-token document and prefill is weak — that's a long TTFT, not a slow stream. Second, throughput ≠ feel for batch work. If you're processing many requests at once, total tokens-per-second across the whole batch can be far higher than the per-request number, because the hardware fills idle time. A server doing 8 requests in parallel may post a big aggregate tok/s while each individual stream looks modest.

// Going deeper

Once the basics click, a few nuances explain most of the surprises people hit when they start measuring seriously.

Decode slows as context grows. As the conversation gets longer, the model must attend to a growing KV cache (the stored keys and values for every previous token). That cache eats memory bandwidth too, so decode speed gradually drops over a long chat. A model posting 40 tok/s at the start of a session may be at 30 tok/s ten thousand tokens in. Benchmarks taken on a short prompt flatter real long-conversation speed.

Quantization buys speed and memory, not magic. Because decode is bandwidth-bound, shrinking the model with quantization speeds up generation roughly in proportion to how much smaller it gets — and lets bigger models fit in VRAM in the first place. The trade is a small quality cost at aggressive levels. This is why the GGUF format ships a model in many quant sizes; you pick the largest one that fits and still hits your target tok/s.

Batching boosts throughput, not single-stream speed. Serving engines built for many users (like vLLM) process multiple requests together, which raises aggregate tokens per second dramatically because the weights, read once from memory, serve several sequences at the same time. For a single local user typing one message, batching does nothing — your decode speed is what it is.

Watch out for thermal throttling and offloading. A laptop that benchmarks at 25 tok/s cold can sag after a few minutes as it heats up and the chip downclocks. And if part of the model is offloaded to CPU/system RAM because it doesn't fit in VRAM, the slow-memory portion drags the whole decode rate down — sometimes by an order of magnitude. When a number looks mysteriously bad, check whether the model fully fits in fast memory first.

The durable mental model: prefill is compute-bound and fast, decode is memory-bandwidth-bound and is the number you feel. Quote the decode (eval) rate, check memory bandwidth before you buy hardware, keep the whole model in fast memory, and pick a quant size that hits the tok/s your task actually needs — chat, coding, or batch.