Offline vs Online LLM Evaluation: When to Use Each

// In plain English

There are two completely different moments where you can check whether your LLM feature is any good. The first is before you ship: you run the model against a fixed set of test cases you prepared in advance and grade the answers. The second is after you ship: you watch how real users actually behave with the live feature. The first kind is called offline evaluation; the second is called online evaluation.

Think of a new recipe. Offline eval is the test kitchen: you cook the dish ten times against a written checklist — right salt, right texture, plated correctly — in a controlled space where you can repeat the exact same conditions all day. Online eval is opening night at the restaurant: you can't control who walks in or what they order, but you can watch how many plates come back, how many guests reorder, and what the reviews say. The test kitchen tells you the dish is correct. The dining room tells you whether real people actually like it.

Neither one replaces the other. The test kitchen can't predict every order a real customer dreams up, and the dining room is a terrible place to discover your salt was wrong — you've already served it. Mature teams run both: offline to catch problems cheaply before launch, online to catch the problems only reality reveals.

// Why it matters

LLMs are non-deterministic and sensitive to tiny prompt changes. A wording tweak that fixes one case can quietly break five others, and you will not feel it by eyeballing a few examples. You need a way to measure change — and the right way depends on whether you've shipped yet.

Before launch, you can't watch real users — they don't exist yet. Offline evaluation is the only feedback loop available. It lets you compare two prompts, two models, or two retrieval settings against the same fixed questions and get a number you can trust. Change one thing, re-run the suite, see if the score went up or down. That repeatability is the whole point: same inputs, same grader, so any score difference comes from your change, not from luck.

After launch, your test set stops being the truth. Real users ask things you never imagined, in messy phrasing, about edge cases your golden questions never covered. A feature can pass every offline test and still frustrate people — because the offline set was your guess about what users would do, and users rarely cooperate. Online evaluation closes that gap by measuring the only thing that ultimately matters: behavior on live traffic.

• Offline catches regressions cheaply. A bad change scores lower on your suite before it ever reaches a user. The cost of a failure is a red number in CI, not an angry customer.
• Online catches blind spots. It surfaces the questions, languages, and failure modes you didn't think to put in your test set — the 'unknown unknowns' no offline suite can anticipate.
• Together they form a ratchet. Every real failure online becomes a new offline test case, so the same bug can never silently come back. Online finds it once; offline pins it down forever.

// How it works

The two methods run at opposite ends of the lifecycle and consume different inputs. Offline runs in development and CI against a dataset you control. Online runs in production against traffic you don't. Here is the full loop they form together.

Offline: grade a frozen dataset

Offline evaluation runs against a golden dataset — a curated, fixed set of inputs paired with what a good answer looks like. You feed each input to your system, collect the output, and grade it with a checker. The checker might be plain code (does the JSON parse? does the answer contain the required order ID?), or an LLM-as-a-judge scoring more open-ended quality. The output is an aggregate score you can compare across versions. Because the dataset is frozen, re-running it next week measures your code, not a moving target — see how to build an eval suite.

Online: measure real traffic

Online evaluation has no expected answer to grade against — you don't know the 'right' reply to a question you've never seen. Instead it leans on signals that real usage produces: explicit feedback (a thumbs-up or thumbs-down, a copy or a regenerate click), implicit behavior (did the user accept the answer and move on, or rephrase and retry?), business outcomes (did support tickets drop?), and automated guardrails that flag bad outputs as they happen. Crucially, online is where A/B testing lives: route some users to version A and some to version B, then compare their real-world metrics with statistics.

Notice the asymmetry: offline gives you a clean number but on data you chose, while online gives you reality but a noisy, indirect read on quality. A thumbs-down tells you a user was unhappy — it doesn't tell you why, or what the correct answer was. That tradeoff is the heart of choosing between them.

// Offline vs online at a glance

The two methods differ on almost every axis — when they run, what data they use, how fast they are, and what they can and can't see. This table is the one-screen summary worth keeping.

// The handoff: ship after offline, then watch online

In practice you don't choose one — you run them in sequence, and the seam between them is the most important part. The pattern is simple: offline is your gate, online is your monitor.

1. Iterate offline. Build a golden dataset, run your change against it, and tune until the score beats the current baseline. Nothing ships if the offline suite regresses — this is your safety net while iterating fast and cheap.
2. Ship to a small slice. Roll the new version out behind a flag to a fraction of traffic (a canary), often as the 'B' arm of an A/B test against the live 'A' version.
3. Watch online metrics. Compare the two arms on real signals — thumbs-up rate, retry rate, guardrail triggers, downstream business outcomes. Give it enough traffic to be statistically meaningful, not just a hopeful glance at the first hour.
4. Feed failures back. Every clear online failure becomes a new row in the golden dataset. Now your offline suite is smarter than it was, and that exact bug can never slip through again.

A common mistake is treating online metrics as the only gate and skipping offline because 'we'll see it in production.' You will — at the cost of real users hitting the bug, and a slow, noisy A/B test where a clear offline run would have told you in thirty seconds. Offline isn't optional just because online exists; it's the cheap, fast layer that keeps obviously-broken changes from ever reaching the expensive layer.

// Common pitfalls

Most eval mistakes come from expecting one method to do the other's job, or from trusting a signal more than it deserves.

• Treating offline scores as production truth. A 95% pass rate on your golden set says you're good on the cases you chose. It says nothing about the queries you never imagined. High offline scores are necessary, not sufficient.
• Reading online signals as ground truth. A thumbs-down can mean a wrong answer, a slow answer, an answer the user didn't like the tone of, or a misclick. Feedback is sparse and biased — angry users click more than happy ones. Treat it as a hint, not a verdict.
• Calling an A/B test early. LLM metrics are noisy; the first few hours can swing wildly. Stop a test before it reaches statistical significance and you'll 'confirm' whatever the early noise said. Decide your sample size up front.
• A stale golden dataset. If your offline set never grows from real failures, it slowly drifts away from reality and your green CI lies to you. The dataset must be a living thing, fed by online findings.
• No logging, so no online eval at all. You cannot evaluate traffic you didn't capture. If inputs, outputs, and context aren't logged from day one, online evaluation is impossible after the fact.

// Going deeper

Once the basic offline/online split is clear, several refinements matter for serious systems.

Online LLM-as-a-judge. You're not limited to user feedback online. You can run an LLM judge over a sample of live traffic to score quality automatically, giving you a continuous quality metric without waiting for users to click. Sampling keeps the cost bounded — you don't need to grade every request to spot a trend.

Guardrails are real-time online evals. A classifier that blocks toxic or off-topic outputs before they reach the user is evaluation running inline, per request, with the power to act. Unlike offline grading, it must be fast and cheap enough to sit in the response path, which constrains how heavy it can be.

Benchmarks are a third, separate thing. Public benchmarks and Chatbot Arena measure models in general, not your application. They help you pick a base model, but a great benchmark score never proves your specific feature works — that's still your offline and online job. Don't confuse choosing a model with evaluating your product.

The drift problem. Even a feature that launched perfectly can degrade: a model provider updates the underlying model, user behavior shifts, or your data distribution moves. Offline eval can't see drift because its dataset is frozen by design — only continuous online monitoring catches a system that was fine yesterday and isn't today. This is the deepest reason online evaluation never ends: quality is not a state you reach, it's one you maintain.

Where to go next. Solidify the offline side with metrics and the code vs model-graded distinction, then write your first eval to make all of this concrete.