Why Do LLMs Give Different Answers to the Same Prompt?

// In plain English

Ask an LLM the exact same question twice and you may get two different answers. Same prompt, same model, same settings — yet the wording, the structure, sometimes even the conclusion can shift. This isn't a bug, and it isn't the model "changing its mind." It's a property of how these systems generate text, layered on top of how they actually run on real hardware.

Think of asking a panel of experts to write a one-paragraph summary of the same article. Every expert read the same text and broadly agrees on the facts, but each one phrases things a little differently, and if you asked the same expert on Monday and again on Friday, even they wouldn't reproduce their summary word for word. The knowledge is stable; the exact output isn't. An LLM behaves the same way — it samples one of many plausible continuations each time rather than replaying a fixed script.

Determinism means same input, same output, every single time. A pocket calculator is deterministic: 2 + 2 is always 4. LLMs are usually nondeterministic — and, importantly, they can drift even when you turn the most obvious source of randomness all the way off. That last part surprises most people, and it's what this article is really about.

// Why it matters

If you only ever chat casually with an AI, drift is harmless — a slightly different phrasing changes nothing. The moment you build on top of a model, it starts to matter a lot.

• Testing and debugging. If your test sends a prompt and checks the output against a fixed string, it will pass one run and fail the next — through no change in your code. You can't write a normal assertion against a moving target.
• Caching. Many systems cache answers to identical prompts to save money and time. If the model never returns the same thing twice, naive exact-match caching never hits.
• Reproducing bugs. A user reports "the model gave me a wrong, dangerous answer." You paste the exact prompt in to investigate — and get a perfectly good answer. The bad output may be genuinely hard to reproduce, which makes it hard to fix.
• Audits and compliance. In regulated settings (finance, healthcare, law) you may need to show why a system produced a specific output. "It was a bit random" is not a satisfying answer to an auditor.
• Evaluation. When you compare two prompts or two models, run-to-run noise can masquerade as a real difference. You have to measure carefully to tell signal from sampling jitter.

The practical takeaway: you can't eliminate nondeterminism in most setups, but once you understand where it comes from, you can reduce it where it counts and design around it everywhere else. Builders who assume "temperature 0 = fully reproducible" get burned when a pipeline that looked stable in testing starts drifting in production.

// How it works

An LLM generates text one token (a word or word-piece) at a time. At each step it doesn't pick a single "correct" next token — it produces a probability distribution over the entire vocabulary: maybe "sunny" at 60%, "warm" at 25%, "clear" at 10%, and a long tail of less likely options. A separate step, called sampling, then chooses one token from that distribution. The chosen token is appended, and the whole process repeats for the next token.

Nondeterminism can creep in at two very different layers of this loop. The first is deliberate randomness in sampling — the obvious cause. The second is tiny numerical differences in the model's math — the subtle cause that survives even when you remove the first. Keeping these separate is the key idea of the whole article.

Cause 1 — sampling randomness (the obvious one)

The main dial here is temperature. High temperature flattens the distribution so unlikely tokens get a real chance — more variety, more surprise. Low temperature sharpens it toward the top choices. At temperature 0 the sampler is told to always take the single highest-probability token (called greedy decoding), which removes the intentional randomness. Other knobs like top-p and top-k shape which tokens are even eligible before sampling picks one.

Cause 2 — floating-point and batching (the subtle one)

Even at temperature 0, with greedy decoding, you can still see different answers. The reason lives below the model, in how the arithmetic runs on a GPU. Models do their math in floating-point, an approximate way of representing numbers. Floating-point addition is not associative: (a + b) + c can give a slightly different result than a + (b + c). The differences are microscopic — far beyond any decimal you'd normally notice.

But GPUs add up thousands of numbers in parallel, and the order they combine them in can change from run to run depending on how the work gets scheduled. A big driver of this is batching: a hosted model serves many users at once by grouping requests together, and the batch your request lands in (its size, who else is in it) shifts which arithmetic path runs. A microscopic difference in a token's score can be enough to flip which token is the new "highest probability" — and once one token differs, every token after it can cascade onto a completely different path.

So temperature 0 removes the chosen randomness but not the hardware variability. Most of the time the top token wins by a comfortable margin and the output is stable; occasionally two tokens are nearly tied and a rounding difference decides it. That is why "temperature 0" reduces drift dramatically without fully guaranteeing it on shared infrastructure.

// All the sources of drift, ranked

It helps to see every reason an identical-looking request can produce a different answer, from the ones you control to the ones you usually don't. Several of these are not about the model's math at all — they're about the input changing when you didn't realize it.

// Practical steps to maximize reproducibility

You usually can't reach perfect, bit-for-bit determinism against a hosted API, but you can get much closer. Stack these from most to least impactful.

1. Set temperature to 0 (or very low). This removes the single biggest source of variation. It won't be perfect on shared infrastructure, but it's the highest-leverage change.
2. Pin a seed if the provider exposes one. Combined with low temperature, it makes the sampling step reproducible.
3. Pin the exact model version. Use a specific, dated model identifier rather than a floating alias that may upgrade under you.
4. Freeze the entire prompt. Same system prompt, same examples, same ordering — and check for anything your framework injects (timestamps, retrieved chunks, tool definitions).
5. Log inputs and outputs. When you can't guarantee reproducibility, capture the exact request and response so you can investigate later without re-running.
6. Test for behavior, not exact strings. Assert that the answer contains the right facts, matches a schema, or passes a check — not that it equals one fixed paragraph.

Here's the difference in test design. A brittle test breaks on harmless rewording; a robust one survives drift while still catching real regressions.

# Fragile: depends on the model producing one exact string.
# Passes one run, fails the next even with no code change.
assert answer == "The capital of France is Paris."

# Robust: checks the thing you actually care about.
assert "paris" in answer.lower()

# Robust for structured output: validate shape, not wording.
import json
data = json.loads(answer)
assert data["capital"] == "Paris"
assert isinstance(data["population"], int)

// Going deeper

A few finer points separate a working mental model from a precise one.

Temperature 0 is not literally division by zero. Conceptually, sampling divides the model's scores by the temperature before turning them into probabilities, and dividing by zero is undefined. In practice, implementations treat temperature 0 as a special case meaning "always take the most likely token" (greedy decoding). So temperature 0 is a convention for greedy, not a real arithmetic operation.

Greedy is not always the "best" answer. Taking the single highest-probability token at every step is locally optimal but can paint the model into a corner — an early safe choice can lead to a worse overall continuation than a slightly riskier path would have. This is why some systems deliberately keep a little temperature even when they value consistency: the most reproducible output and the best output aren't always the same token sequence.

The batching effect is an active engineering frontier. Making large-model inference batch-invariant — so a request yields the same result regardless of who else it's batched with — is achievable with carefully written GPU kernels that fix the reduction order, but it can cost throughput. Providers weigh that tradeoff differently, which is part of why reproducibility guarantees vary between APIs and even between a hosted API and the same model run locally on one GPU.

Determinism is independent of correctness. A perfectly deterministic model can still confidently produce a wrong answer every time, and a nondeterministic one can be right every time in different words. Pinning down reproducibility helps you test, audit, and debug — it does nothing on its own to make answers more accurate. For why models produce confident-but-wrong content in the first place, see why LLMs hallucinate.

Where to go next: tune the obvious dial with temperature, understand the eligibility filters with top-p vs top-k, and revisit the token-by-token generation loop in how LLMs work. Together they explain not just that an LLM drifts, but exactly which lever moves which part of the behavior.