Context Window vs Training Data: What Does a Model Actually Know?

// In plain English

An LLM has two completely separate sources of knowledge, and mixing them up causes endless confusion. The first is everything it learned during training — patterns, facts, and relationships absorbed from billions of documents and compressed into the model's billions of parameters. The second is everything you put in the prompt right now — the text in the context window. These two things look the same from the outside (both produce answers), but they work very differently inside.

Here is a concrete analogy. Imagine hiring a doctor who went to medical school ten years ago and hasn't read a single paper since. She has enormous knowledge baked into her head from her education — anatomy, drug interactions, diagnosis patterns. That's the training data. Now you hand her today's patient chart: current symptoms, recent test results, this week's medication list. That's the context window. She'll use both. But the patient chart is the only place she can learn anything new in this visit. The moment she walks out of the room, she forgets everything on the chart. Her medical-school knowledge stays forever; the chart knowledge evaporates.

In AI research these two kinds of knowledge have names. Parametric knowledge is what lives in the model weights — the numbers that define the model, frozen after training. Contextual knowledge (sometimes called non-parametric knowledge) is the text in the context window right now. Parametric knowledge persists across every conversation; contextual knowledge lasts only for that single request.

// Why it matters

This distinction isn't academic — it is the engineering bedrock that every serious AI product is built on. The moment you understand it, several previously mysterious behaviors click into place:

• Outdated answers — The model confidently states something that was true two years ago but has since changed. That's parametric knowledge talking. Its training data has a cutoff date; the world kept moving.
• "I don't know about that" for recent events — If something happened after training ended, it isn't in the weights. The model can't know what it never saw.
• Hallucinated but plausible facts — The model fills gaps in its parametric knowledge by pattern-completing to something plausible but fabricated. It doesn't have a "I don't know this for certain" switch.
• Correct answers when you paste context — Provide the right document in the prompt and the model suddenly answers correctly, even for facts it had wrong before. The contextual knowledge overrides (or supplements) the parametric guess.

The practical punchline: you cannot rely on a model's parametric knowledge for anything time-sensitive, private, or highly specific. Company policies, yesterday's stock price, your customer's account status, proprietary research — none of that is in the weights. You have to supply it in the context window. This is the exact problem that Retrieval-Augmented Generation (RAG) was designed to solve: instead of hoping the model knows, you fetch the relevant document and paste it in.

For developers building with LLMs this split creates two distinct engineering levers. You control contextual knowledge completely: whatever you put in the prompt is what the model sees. Parametric knowledge you can only influence by choosing a better-trained model or running a fine-tuning job — neither is a quick fix. The practical guideline: default to putting facts in the context window; fall back to fine-tuning only for style, format, and behavior, not for knowledge.

// How it works

When a model generates an answer, it draws on both sources simultaneously. Understanding how each one is encoded helps you predict when to trust it and when to override it.

Parametric knowledge: facts frozen in weights

Training is a process of showing the model an enormous corpus of text — web pages, books, code, papers — and adjusting billions of floating-point numbers (the weights) so the model gets better at predicting the next token. After trillions of such adjustments, the weights have implicitly encoded an enormous amount of world knowledge. There is no lookup table; the facts are distributed across the weights in a way even researchers don't fully understand. Asking the model who wrote Hamlet is not a database query — it's a forward pass through a neural network that statistically tends to produce "Shakespeare" because of patterns it saw millions of times.

Once training ends, the weights are frozen. Every user, every conversation, every request hits the exact same set of numbers. Nothing you say changes those numbers. The knowledge cutoff is baked in.

Contextual knowledge: facts in the prompt

The context window is a flat sequence of tokens — your system prompt, the conversation history, any documents you attached, and the model's previous replies all concatenated together. The model reads this sequence from scratch on every single request (it has no memory between calls) and uses it alongside the parametric knowledge to generate its response. Because the context is explicit text that the model can directly attend to, it tends to override conflicting parametric knowledge — though research shows this is not guaranteed when the conflict is sharp.

// Why RAG exists: bridging the gap

Retrieval-Augmented Generation (RAG) is the direct engineering response to the parametric-vs-contextual split. The premise is simple: a model's weights cannot be updated cheaply or quickly, but the context window can be filled with anything at any time. So instead of hoping the model knows the answer, a RAG system retrieves the answer from an up-to-date external source and puts it in the prompt. The model then reads the document and answers from contextual knowledge rather than parametric memory.

A typical RAG pipeline looks like this:

1. The user asks a question.
2. A retriever (usually a vector-similarity search over an embedding index) finds the most relevant documents or passages from an external store.
3. Those passages are inserted into the prompt, before the user's question.
4. The model generates an answer based on the retrieved passages — contextual knowledge — rather than relying on parametric memory.
5. The retrieved context is discarded after the response; the model's weights are untouched.

The external store can be updated at any time — new documents indexed, old ones revised — and the model will immediately answer from the freshest version on the next request. No retraining required. This is what makes RAG so practical for enterprise use cases: internal wikis, product documentation, legal databases, customer support knowledge bases. All dynamic, none of it in a model's weights.

One important limitation: RAG doesn't make the context window infinite. The retrieved passages still have to fit alongside the rest of the prompt. Production RAG systems carefully budget their context space — a typical setup might reserve a few thousand tokens for retrieved chunks, leaving room for the system prompt, conversation history, and output. As context windows have grown to around a million tokens in frontier models like Claude and Gemini, the retrieval step has gotten more forgiving — but the fundamental constraint never disappears.

// Common pitfalls and misconceptions

Several persistent misconceptions follow from not fully internalizing the parametric-vs-contextual split:

The knowledge-cutoff clock

Every production model has a training cutoff: the date beyond which no new data was included. Events after that date simply do not exist in the weights. This matters more than people realize: major models are often trained on data that is six to eighteen months old by the time they ship, and then deployed for another year or two. A model you're using today might have parametric knowledge that ends two or three years ago. For anything time-sensitive, assume the weights are stale and supply current information in context.

// Going deeper

When weights and context conflict. Research published in 2024-2025 (arxiv.org/abs/2410.08414) found that when retrieved context directly contradicts what a model believes from training, LLMs tend to resolve the conflict in favor of parametric knowledge — the opposite of what RAG engineers want. This has spawned a sub-field: techniques like parametric-knowledge reinforcement, explicit grounding instructions, and chain-of-thought prompting that asks the model to cite the passage it relied on. For high-stakes applications, always instruct the model to answer strictly from the provided context, and consider using citations as a verification mechanism.

Attention heads process them separately. Recent interpretability research suggests that contextual and parametric knowledge are processed by different attention heads with relatively little cross-type competition. This is encouraging: it means a well-prompted model can integrate both types without one completely drowning out the other. But it also implies that architectural choices — which layers handle retrieval, how attention patterns are shaped — will matter increasingly as context windows grow longer.

Fine-tuning injects behavior, not reliable facts. It's tempting to fine-tune a model on a company's internal documents and expect it to recite those facts accurately. In practice, factual recall from fine-tuning degrades under distribution shift — ask the same question slightly differently and the model may hallucinate. This is because fine-tuning updates weights in a diffuse, distributed way, not like writing a row to a database. Facts that need to be recalled with high reliability still belong in the context window, not in the weights.

The emerging middle ground: long-context RAG. As context windows have expanded to one million tokens, some teams now load entire knowledge bases into a single prompt and skip vector search altogether — "full-document RAG". This sidesteps retrieval errors but shifts costs to inference: a million-token prompt is expensive and slower. The field is actively exploring when dense retrieval, sparse keyword search, and brute-force long-context loading are each the right tool, and hybrid approaches are increasingly common.

The fundamental asymmetry. Contextual knowledge is cheap to update, free to customize, and can be as fresh as the moment you build the prompt. Parametric knowledge is expensive to update (requires training runs), cannot be personalized per user, and ages constantly. This asymmetry is not a bug — it is the design space that almost every LLM application architecture navigates. Understanding it at a deep level is the prerequisite for making sensible decisions about when to reach for RAG, when to reach for fine-tuning, when to reach for a newer base model, and when the answer is just a better-written prompt.