Why Are LLMs Bad at Math? The Limits of Next-Token Prediction

// In plain English

A large language model does not calculate. When you ask it 27 x 14, it does not multiply. It predicts the next most likely token, one at a time, based on patterns it saw in billions of pages of text. Multiplication tables, common sums, and worked examples appear so often in that text that the model learned to imitate the answer. It is recalling what arithmetic usually looks like, not doing the arithmetic.

Picture a brilliant student who memorized the entire multiplication chart and read thousands of math worksheets, but was never taught the rules for carrying and borrowing. Ask them 7 x 8 and they answer 56 instantly, because they have seen it a thousand times. Ask them 4837 x 219 and they freeze, because that exact problem was not on any worksheet they memorized. So they write down a number that looks about right, with confidence, and move on. That is an LLM doing math.

This is why the failure feels so strange. The same model that writes correct, complex code can confidently tell you that 35.8 - 12.95 = 23.85 (wrong; it is 22.85) and never flinch. It is not lying and it is not broken. It is doing exactly what it was built to do: produce a fluent, plausible continuation. Plausible and correct are not the same thing, and arithmetic is the place where that gap shows up most clearly.

// Why it matters

If you are building anything on top of an LLM, this single fact changes your design. A model that is usually right at arithmetic but silently wrong sometimes is worse than one that is reliably wrong, because you cannot tell the difference by looking at the output. The answer is always confident and well-formatted.

• Anything with numbers in it is at risk. Invoices, financial summaries, unit conversions, dosage calculations, date math, percentages, splitting a bill. If your product asks the model to add, multiply, or convert numbers in its own head, you have a quiet bug waiting to surface on the one input that did not match a memorized pattern.
• The errors are unpredictable, not random. The model is fine on small, common numbers and degrades on long, unusual, or multi-step problems. So it passes your quick demo (2 + 2, 10% of 50) and then fails in production on 1,284,930 / 37 — the exact case you did not test.
• Confidence hides the failure. A wrong answer comes wrapped in the same calm, authoritative tone as a right one. There is no built-in uncertainty signal saying "I guessed here." This is closely related to why models hallucinate facts — same root cause, different symptom.

The lesson is not "LLMs are useless for math." It is "don't make the model the calculator." Once you know why raw arithmetic is unreliable, the correct architecture becomes obvious: let the model decide what to compute and hand the actual computation to something that computes exactly. Builders who understand this ship reliable products; builders who assume the model can just "do the math" ship subtle errors into spreadsheets and dashboards.

// How it works

To see why arithmetic is hard, follow what actually happens inside the model when you type a sum. There are two separate problems stacked on top of each other: how the model sees the numbers, and how the model produces its answer.

Problem 1: the model predicts, it doesn't compute

An LLM generates text by repeatedly answering one question: given everything so far, what token is most likely to come next? It learned those probabilities from training text. For arithmetic, that means the model only knows the answer if the pattern 7 x 8 = ? and its result appeared often enough in training for the next-token probability of 56 to be high. There is no multiplication circuit being run. There is only "in text like this, the number that usually follows is…"

This is why common sums are nearly always right and large or unusual ones go wrong. 12 x 12 is everywhere in the training data, so the model effectively memorized it. 6391 x 472 almost certainly never appeared as an exact string, so the model has no memorized answer to recall — it produces a number with the right shape (about the right number of digits) but wrong value.

Problem 2: tokenization chops numbers apart

The model never sees digits the way you do. Text is first broken into tokens by a tokenizer, and numbers get split in inconsistent, place-value-blind ways. A number like 1457 might become one token, or two tokens like 14 and 57, or three — depending on the tokenizer and what is around it. The model does not reliably see the digit in the "hundreds place"; it sees arbitrary chunks.

Human arithmetic depends on lining up columns: ones under ones, tens under tens, carrying left. If the model cannot reliably tell which chunk is the tens place and which is the thousands place, column-by-column addition is almost impossible to do in its head. This tokenization quirk is the same reason models miscount letters in a word — the famous strawberry problem, where a model insists "strawberry" has two r's because it never saw the individual letters, only token chunks.

The two problems compound

Put them together: the model is trying to recall a memorized answer it doesn't have, for a number it can't even see cleanly, while generating one token at a time with no scratch pad. It is genuinely surprising it gets any multi-digit math right. It manages because so much arithmetic and so many worked examples sit in its training data that it learned rough procedures — but rough procedures break on the long tail of inputs.

// A worked example: watch it fail, then fix it

Take a single problem and watch how the way you ask changes whether the model gets it right. The math never changes; only the model's working conditions do.

Ask for the answer directly

User: What is 4837 x 219? Reply with only the number.
Model: 1059303

The real answer is 1,059,303… actually let us not trust either of us — 4837 x 219 = 1,059,303. The point is that forcing a one-token-shaped answer gives the model zero room to work. It has to emit the result immediately, so it falls back on a memorized-shape guess. On numbers this size it is often slightly off, and you cannot tell from looking.

Let it show its work (chain-of-thought)

User: What is 4837 x 219? Work through it step by step.
Model: 4837 x 219 = 4837 x 200 + 4837 x 19
       4837 x 200 = 967400
       4837 x 19  = 91903
       967400 + 91903 = 1059303

By breaking the problem into smaller sums it has seen, the model turns one hard prediction into several easy ones. Each intermediate step (4837 x 200) is closer to a memorized pattern, and the final addition is simpler. This is chain-of-thought prompting: the extra tokens are the model's working memory. It is more reliable than a direct answer — but still not guaranteed, because every step is still a prediction, not a computation.

Give it a real calculator (tool use)

# The model decides WHAT to compute and emits a tool call;
# your code runs the actual arithmetic and returns the exact result.
def calculator(expression: str) -> str:
    # A sandboxed, deterministic evaluator — never the LLM.
    return str(safe_eval(expression))

# Model output (a tool call, not a guessed number):
#   calculator("4837 * 219")
# Tool returns: "1059303"  <- exact, every time
#
# The model then writes the final sentence around that exact value.

// The fixes, ranked

There are four common ways to make an LLM trustworthy with numbers. They trade off effort, cost, and reliability. Pick based on how much a wrong number actually costs you.

Reasoning models deserve a note. Newer reasoning models are trained to generate a long internal chain of thought before the final answer, so they effectively do chain-of-thought automatically and self-correct more. They are markedly better at math than older models. But they are still predicting tokens — slower and pricier, and not exact. For real arithmetic that must be correct, they still belong behind a tool.

The hierarchy is simple: if the answer must be exactly right, route the computation to code. Use chain-of-thought or a reasoning model when you need the model to figure out which numbers to compute and in what order, then let deterministic code do the computing. The model's job is to translate a messy request into a precise calculation; the calculator's job is to be precise.

// Going deeper

Once the basics click, a few sharper points are worth knowing — both about the limits and about where the field is heading.

Digit tokenization is an active design choice. Some model builders now force numbers to be tokenized one digit at a time, or right-to-left in fixed groups, specifically to preserve place value. This measurably improves arithmetic. It is a reminder that "LLMs are bad at math" is partly a tokenizer decision, not a law of nature — change how numbers are chopped up and the failure rate moves.

Length generalization is the deep problem. A model can learn to add 3-digit numbers from training and still fail at 10-digit numbers it never saw. True arithmetic generalizes to any length because it follows a rule; pattern-matched arithmetic only covers the lengths it practiced. This gap — performing a learned procedure on inputs longer than any seen in training — is one of the genuinely hard open questions in the field, and it is why benchmarks quietly cap the digit counts.

"Good at math" can mean two different things. A model can be strong at mathematical reasoning (setting up the right equation for a word problem, choosing the right approach) while being weak at raw computation (executing the arithmetic). These are separate skills. Reasoning models and chain-of-thought boost the first; only tools fix the second. When you read that a model "scored well on a math benchmark," check which kind of math — graded proofs reward reasoning, while exact-answer arithmetic rewards a calculator.

Where to go next. The root cause here — fluent but unverified output — is the same machinery behind why LLMs hallucinate in general, and it traces back to how LLMs work at the token level. If you want the tokenization angle in depth, see how tokenization works and the strawberry problem. The durable takeaway: an LLM is a reasoning and language engine, not an arithmetic unit. Build it to delegate exact math, and it becomes reliable; ask it to do exact math from memory, and it will eventually, confidently, hand you a wrong number.