What to Do When Your Prompt Is Too Long for the Context Window

// In plain English

Every language model reads and writes inside a fixed budget of tokens — its context window. That budget has to cover everything: your system prompt, the documents you paste in, the whole back-and-forth of a chat, and the space the model needs to write its reply. When the total crosses the limit, you have context window overflow — the prompt simply does not fit.

Think of the context window as a single whiteboard the model uses for one request. You write the instructions, the reference material, and the conversation on it, and the model needs room left over to write its answer. A small whiteboard is fine for a quick note. But try to fit a 300-page contract, a year of chat history, and a long question, and you run out of board. Something has to come off — the only real question is what, and how you choose.

This article is not about what a context window is — that's covered in context windows. It's the practical sequel: your input is too big, the API is about to reject it (or quietly drop the front of it), and you need a plan. There are four standard moves — truncate, summarize, chunk-and-map, or retrieve — and the skill is knowing which one fits your situation.

// Why it matters

Overflow is not a rare edge case — it is the default failure mode of any real application the moment inputs get bigger than your test cases. A chatbot works beautifully in a demo with three messages, then breaks in week two when a power user's conversation reaches a hundred turns. A document tool handles a memo, then chokes on a merger agreement. If you build with LLMs, you will hit this.

What actually happens at the limit depends on the API, and the two behaviors are dangerously different:

• Hard rejection. Most chat APIs return an error like "this request exceeds the maximum context length" and refuse to run. Annoying, but honest — you know immediately that you have a problem to solve.
• Silent truncation. Some tools and middleware quietly drop the oldest or front part of the prompt to make it fit, then run anyway. This is worse, because it can throw away your system prompt or the very document the user asked about — and you get a confident, wrong answer with no error at all.

There is also a quieter cost. Even when a giant prompt does fit, it is slow and expensive — you pay for every input token on every call — and models recall facts buried in the middle of a long context less reliably than facts near the edges (the well-known "lost in the middle" effect). So managing context length is not only about avoiding errors; it's about keeping answers fast, cheap, and accurate. That broader discipline is context engineering; overflow handling is its most concrete, unavoidable piece.

// How it works: the four moves

Every fix for overflow does one of two things: it makes the input smaller before it reaches the model, or it changes the strategy so the full input never has to be in the window at once. Here are the four standard moves, from simplest to most powerful.

1. Truncate — just cut it down

The bluntest fix: drop tokens until the prompt fits. Keep the most important parts (usually the system prompt and the latest user message) and cut the rest — the oldest chat turns, or the tail of a document. Truncation is instant, free, and adds no extra model calls. Its weakness is equally obvious: whatever you cut is gone, so if the answer lived in the part you dropped, the model can't find it. Use truncation when the cut material is genuinely low-value (old small talk, boilerplate) — never when it might hold the answer.

2. Summarize — compress instead of cut

Instead of deleting the old material, replace it with a shorter version of itself. The classic case is a long chat: when history grows too big, you ask the model to write a compact summary of the older turns, then keep that summary plus the most recent messages verbatim. This is often called a rolling summary or conversation memory. You preserve the gist of a long history in a fraction of the tokens. The cost is an extra LLM call to produce each summary, and some loss of detail — a summary is lossy by definition, so specific names, numbers, or quotes can blur.

3. Chunk-and-map — process in pieces

When the task is over the whole input — "summarize this 200-page report," "find every clause about liability" — you can't just throw most of it away. Instead, split the input into chunks that each fit the window, run the model on every chunk, then combine the results. This is the map-reduce (or map-and-combine) pattern: map the task across chunks, then reduce the partial answers into one final answer. It handles inputs of any size, but it's slower and pricier (one call per chunk plus a combine step), and reasoning that spans two distant chunks can get lost because no single call sees both.

4. Retrieve — only fetch what's relevant

The most scalable move flips the problem around. Instead of trying to fit a huge corpus into the prompt, you search it and insert only the handful of passages the question actually needs. This is retrieval-augmented generation: index your documents once, then at question time pull the top few relevant chunks into the prompt. The window stays small no matter how big the underlying library is — you could have a million documents and still send only 2,000 tokens. The trade-off is real infrastructure (an index or vector store) and the risk that retrieval misses the right passage.

// Worked example: a rolling chat memory

The most common place builders meet overflow is a chat that just keeps growing. Here's the standard pattern in plain Python: keep recent turns verbatim, and once history crosses a token budget, fold the older turns into a running summary. The key idea is a token budget check before every call, not after the error.

from anthropic import Anthropic

client = Anthropic(api_key="sk-ant-...")
MODEL = "claude-sonnet-4-6"

summary = ""            # running summary of older turns
recent: list[dict] = []  # last few verbatim messages
TOKEN_BUDGET = 8000     # max tokens we allow for history

def token_count(messages):
    # Use the provider's real token counter, not a guess.
    r = client.messages.count_tokens(model=MODEL, messages=messages)
    return r.input_tokens

def maybe_compress():
    global summary, recent
    # If recent history is over budget, summarize the oldest half.
    while token_count(recent) > TOKEN_BUDGET and len(recent) > 2:
        old, recent = recent[:2], recent[2:]
        text = "\n".join(f"{m['role']}: {m['content']}" for m in old)
        prompt = (
            f"Update this running summary with the new turns.\n\n"
            f"Summary so far:\n{summary}\n\nNew turns:\n{text}"
        )
        summary = client.messages.create(
            model=MODEL, max_tokens=400,
            messages=[{"role": "user", "content": prompt}],
        ).content[0].text

def chat(user_msg):
    recent.append({"role": "user", "content": user_msg})
    maybe_compress()
    system = f"Conversation summary so far:\n{summary}" if summary else ""
    reply = client.messages.create(
        model=MODEL, max_tokens=600, system=system, messages=recent,
    ).content[0].text
    recent.append({"role": "assistant", "content": reply})
    return reply

// Choosing a strategy and avoiding traps

Pick by what kind of input you have and what the task needs. Use this as a quick decision guide:

These moves also combine. A serious assistant might retrieve documents, summarize old chat turns, and truncate a stray oversized field — all in the same request. The strategies are tools, not religions.

Common pitfalls

• Counting tokens with the wrong tool. Guessing from word count, or using a different model's tokenizer, leads to off-by-thousands errors. Use the provider's own token counter on the exact model.
• Forgetting the output budget. The reply lives in the same window. If you fill it to the brim with input, the model has no room to answer and gets cut off mid-sentence.
• Truncating the system prompt. Naive "drop the front" logic can delete your instructions, since the system prompt sits first. Always protect it explicitly when trimming.
• Chunking that splits mid-thought. Cutting a sentence, table, or clause in half garbles meaning. Split on natural boundaries (paragraphs, sections) and overlap a little between chunks.
• Summary drift. Re-summarizing a summary over and over slowly loses fidelity. Keep recent turns verbatim and only compress what's genuinely old.

// Going deeper

Once the four basic moves are second nature, a few more advanced ideas are worth knowing.

"Just use a bigger window" is not a full answer. Modern models offer huge windows — hundreds of thousands or even a million tokens. That genuinely removes overflow for many tasks, and for a single handbook "just paste it" can be the right, simple call. But a bigger window doesn't make every token free or every fact equally recalled: you still pay per input token on every call, latency still grows, and the lost-in-the-middle effect still bites. Overflow handling becomes a cost and quality decision long before it stops being a will-it-fit one.

Prompt caching changes the math. If a large, stable chunk of your prompt (a long system prompt, a fixed document, few-shot examples) repeats across many requests, several APIs let you cache it so you don't re-pay full price to re-process it each time. That makes "keep a big stable context" far cheaper, shifting some decisions away from aggressive trimming. It doesn't raise the window limit — it lowers the cost of reusing what's in it.

Map-reduce has variants. The plain version maps independently then combines once. A refine loop instead carries a running answer through the chunks one at a time, updating it as it goes — better for tasks where later chunks should adjust earlier conclusions, at the cost of being sequential and slower. Choose based on whether your chunks are independent or build on each other.

Structure helps the model use a full window. When you do send a lot of context, clear boundaries matter: label sections, fence documents, and separate instructions from data so the model knows what's what — see structuring prompts with XML or Markdown and putting documents in prompts. A well-organized large prompt is recalled better than a wall of undifferentiated text.

The durable takeaway: overflow is not a bug to patch once, it's a constraint to design around. Decide before you hit the limit which content is load-bearing and which is disposable, measure real token counts, and keep the window full of only what the current answer actually needs.