What Is In-Context Learning? How LLMs Learn Without Retraining

// In plain English

Imagine handing a colleague a few solved examples — "here is an English sentence, here is its French translation" — and then giving them a new sentence to translate. They don't need a language course; the examples in front of them are enough context to figure out the pattern. In-context learning (ICL) is exactly that, except the colleague is a frozen language model and the examples live inside the prompt.

The model's weights are never touched. No gradient is computed. No training loop runs. The model simply reads the examples in the same way it reads any text — left to right, one token at a time — and uses what it sees to produce the right kind of output for the new input that follows. From the outside it looks like the model "learned" the task on the fly. Under the hood, nothing changed.

This ability first surprised researchers when GPT-3 (175 billion parameters) was published in 2020. Previous smaller models could barely use a few examples; GPT-3 could translate, classify, reason, and generate code purely from prompt examples — no fine-tuning required. The OpenAI paper that introduced GPT-3, Language Models are Few-Shot Learners, named this capability and kicked off a decade of research trying to explain it.

// Why it matters for builders

Before ICL, adapting a language model to a new task meant fine-tuning: collect labeled data, pay for a GPU run, version the resulting checkpoint, and repeat whenever the task evolved. That cycle takes hours to days and real money. ICL collapses the iteration loop to seconds — write a few examples in the prompt, test, refine, ship.

This matters practically in several ways:

• Prototyping speed. You can test whether a model can handle a niche classification task in minutes by dropping in three labeled examples. No infrastructure needed.
• Task switching at runtime. The same model endpoint can behave like a sentiment classifier, a JSON extractor, or a SQL generator depending solely on the examples you prepend to the prompt.
• No labeled data requirement at scale. Many real-world tasks have only a handful of annotated examples. ICL can leverage those without needing hundreds or thousands.
• Customization without model ownership. If you are calling a third-party API like Claude or GPT and cannot fine-tune the hosted model, ICL is often your only lever for steering output format and style.

ICL is not a replacement for fine-tuning in every scenario, but it is the right default starting point — especially when requirements are uncertain or changing.

// How it works mechanically

When a transformer reads a prompt, every token attends to every earlier token via the self-attention mechanism. In a few-shot prompt, the model sees the input-output pairs (the "shots") and then the test input, all as one contiguous sequence of tokens. Attention heads can learn to spot the structural pattern across the shots — "input maps to output" — and apply the same transformation to the trailing test input.

Researchers have found that certain attention heads, sometimes called induction heads, specialize in detecting and copying patterns across the context. When these heads see [A][B] ... [A] they predict [B] should follow. In practice this primitive — copy the completion that followed a matching prefix — is the low-level building block that enables complex task recognition from examples.

A striking theoretical result from Akyürek et al. (2022) and related work shows that a linear self-attention layer can implement exactly one step of gradient descent on the examples in its context. This means ICL may be simulating a mini training loop inside the forward pass — but it is a single step, not the hundreds of steps real fine-tuning takes. The equivalence also only holds under simplified linear assumptions; real multi-layer transformers are more complex and the relationship is approximate.

// The three flavors: zero-shot, few-shot, and many-shot

In practice ICL covers a spectrum depending on how many examples you include. The right choice depends on your task complexity, the model's context window, and how much example data you have.

The many-shot regime became practical in 2024 when models with million-token context windows shipped. Google DeepMind's Many-Shot In-Context Learning paper (NeurIPS 2024) demonstrated that scaling to hundreds of examples can match fine-tuning performance on many benchmarks. The same paper showed Gemini 1.5 Pro learning to translate English into Kalamang — an extremely low-resource language — using roughly 400 parallel sentences and a grammar reference provided entirely in-context.

Chain-of-thought (CoT) prompting is ICL applied to reasoning. Instead of showing input → answer, you show input → reasoning steps → answer. The model then generates its own chain of reasoning before committing to an answer. CoT dramatically improves accuracy on math, logic, and multi-step tasks — it is ICL with a richer output structure in the examples.

# Few-shot sentiment classification

Review: "Battery life is terrible."
Sentiment: Negative

Review: "Fastest laptop I have ever owned."
Sentiment: Positive

Review: "Screen is gorgeous but the keyboard feels cheap."
Sentiment: Mixed

Review: "Ships faster than expected and works perfectly."
Sentiment:

// In-context learning vs fine-tuning

Both ICL and fine-tuning adapt a model to a task. They differ in where the adaptation lives and at what cost.

A useful rule of thumb: start with ICL, move to fine-tuning when you have clear, stable requirements and ICL quality has plateaued. Research from VentureBeat-covered studies in 2024 found that in data-matched settings ICL sometimes generalizes better than fine-tuning, particularly on tasks requiring logical deduction from provided context — because fine-tuning can overfit the training distribution while ICL stays flexible. The trade-off flips when you need consistent, low-variance output at scale.

// Where ICL hits limits

ICL is powerful but brittle in ways that catch builders off guard:

• Order sensitivity. The same eight examples in a different order can swing accuracy by 10–30 percentage points. Models can be overly influenced by the last example before the test input. Mitigate by shuffling examples and averaging, or by using retrieval-based selection.
• Lost-in-the-middle effect. In long prompts, models attend more strongly to tokens at the beginning and end of the context. Examples buried in the middle of a large few-shot block may contribute less signal than examples near the query.
• Context window ceiling. You can only fit so many examples before hitting the model's token limit. For tasks that need hundreds of examples to reach useful accuracy, you may eventually need fine-tuning regardless.
• Label-format sensitivity. If one of your few-shot examples uses a slightly different output format — a comma instead of a newline, or Yes instead of yes — the model's output format can become inconsistent. Uniformity in examples matters more than it might seem.
• No persistence across calls. Every API call starts fresh. ICL has no memory; it cannot accumulate knowledge from prior interactions the way a fine-tuned model or a retrieval system can.
• Emergent ability threshold. ICL quality scales with model size. Small models (under ~7 billion parameters) often fail to generalize from examples reliably. If you are using a smaller model, zero-shot with a very explicit system prompt sometimes outperforms few-shot.

// Going deeper

The theoretical explanation for ICL remains an active research frontier. Several accounts compete and are likely all partially correct:

1. Implicit Bayesian inference. Xie et al. (2021) proposed that ICL is equivalent to Bayesian inference: the model has a prior over tasks from pretraining, and the in-context examples update a posterior over which task is being asked. This explains why ICL quality scales with the diversity of pretraining data — a richer prior makes inference sharper.
2. Implicit gradient descent. The linear-attention result mentioned earlier (Akyürek et al., 2022) showed that self-attention can implement one step of gradient descent. Several follow-up papers (Oswald et al., 2023) confirmed the connection in deeper networks, though the equivalence weakens under realistic conditions.
3. Induction heads. Mechanistic interpretability research by Anthropic and others identified specific attention head circuits that copy patterns from earlier in the context. These heads appear to be the proximate mechanism for the in-context pattern-matching — and they emerge as a phase transition during training as model scale increases.
4. Task vector retrieval. Some work frames ICL as the model retrieving a "task vector" — a direction in activation space that encodes the task — and applying it to the query. This framing connects ICL to representation engineering and activation steering.

Why ICL is not real learning. Despite the name, ICL does not update the model's knowledge. Once the conversation ends, nothing is retained. A model that performs perfect sentiment classification via ICL in session A has exactly the same weights in session B and starts from zero again. Compare this to a student who reads a grammar textbook and remembers the rules afterward. The model is more like a student who must re-read the textbook at the start of every exam.

Retrieval-augmented ICL. Rather than hand-picking few-shot examples, production systems often retrieve the most relevant examples from a database at query time — selecting whichever labeled samples are most semantically similar to the current input. This approach, sometimes called retrieval-augmented ICL or dynamic few-shot prompting, can match or exceed static few-shot accuracy while keeping prompt length under control. It is a natural bridge between pure ICL and retrieval-augmented generation (RAG).

ICL as a window into model capability. Because ICL requires no fine-tuning, it is the cleanest way to probe what a model already knows. If a model fails at a task even with eight crystal-clear examples, fine-tuning is unlikely to fix the underlying gap — the capability is simply not in the weights. ICL failure is therefore a useful diagnostic before committing to an expensive training run.