What Is Prompt Compression? Fitting More Into Fewer Tokens

// In plain English

Every time you call a language model, you pay by the token — a token is roughly three-quarters of a word. The prompt you send (your instructions, the retrieved documents, the chat history) is the input, and you are billed for every token of it on every single call. A long, padded prompt is slower and more expensive than a tight one, even when most of the extra words add nothing.

Prompt compression is the practice of shrinking that input — cutting filler, summarizing old material, and removing duplicates — so the model receives the same useful information in far fewer tokens. The goal is not to make the prompt shorter for its own sake; it is to keep everything the model actually relies on while dropping everything it doesn't.

Think of packing a carry-on bag for a trip. You don't leave behind the things you need — your passport, your charger, one outfit per day. You leave behind the bulky packaging, the duplicate chargers, and the "just in case" items you'll never touch. A well-compressed prompt is a well-packed bag: everything essential, nothing wasteful, and it fits through the gate without an extra fee.

// Why it matters

Token count is not a cosmetic detail. It drives three things a builder cares about directly: cost, latency, and quality. Compression improves all three at once.

• Cost. Input tokens are billed per call. A chatbot that resends a 4,000-token system prompt and a growing history on every turn can spend most of its budget re-paying for text that never changed. Trimming the prompt cuts that bill straight away.
• Latency. A model has to read the whole input before it writes the first word of output. More input tokens means more time spent reading. A shorter prompt visibly lowers the time-to-first-token, which users feel as responsiveness.
• Quality. This surprises people: a tighter prompt is often a more accurate prompt. Models recall facts buried in the middle of a long context less reliably than facts near the edges — the "lost in the middle" effect. Padding the prompt with marginal text adds noise that can bury the one sentence that mattered.
• Fitting the window. Every model has a maximum context window. When a long chat or a big pile of retrieved documents approaches that limit, compression is what lets the important parts still fit.

Who should care? Anyone running an LLM feature at scale. A support bot handling thousands of conversations a day, a RAG system stuffing retrieved passages into the prompt, a long-running agent whose history keeps growing — each of these multiplies wasted tokens across millions of calls. At one request a day, compression barely registers. At a million requests a day, it is the difference between a sustainable product and a runaway invoice.

// How it works

A prompt is built from several parts: a system prompt, maybe some few-shot examples, retrieved documents, the conversation history, and the user's latest message. Compression works by examining each part and asking: is this earning its tokens? The techniques fall into three families.

1. Manual compression: write it tighter

The cheapest, most reliable wins come from editing the prompt by hand. Models do not need polite, verbose English — they parse terse instructions just as well. So you cut hedging and filler ("Please could you kindly take a moment to…" becomes ""), collapse repeated rules stated three different ways into one, and remove examples that teach the model nothing new. You also dedupe: if the same policy appears in both the system prompt and a retrieved document, keep it once.

2. Summarization: replace the long with the short

When the content itself is long — a 30-turn chat history, a 10-page document — you replace it with a shorter version that preserves the facts the model needs. The common pattern in a conversation is a rolling summary: keep the last few turns verbatim (recent detail matters), and replace everything older with a compact summary produced by a cheaper model. The summary carries the gist; the recent turns carry the precision.

3. Automatic compression tools

Some tools compress text token-by-token without an LLM rewriting it. The best-known is LLMLingua (from Microsoft Research). It uses a small language model to score how much each token contributes to meaning, then drops the lowest-value tokens — articles, redundant words, predictable filler — leaving a denser prompt that the big model can still understand. The output often looks slightly garbled to a human but still carries the signal the model needs. These tools shine on long, repetitive contexts where manual editing isn't practical.

// A worked example

Here is the same instruction before and after manual compression. Both tell the model exactly the same thing; the second costs roughly half the tokens.

You are a very helpful and friendly customer support assistant.
Please make sure that you are always polite and courteous at all
times. When a customer asks you a question, I would like you to
do your best to answer it accurately. If you happen to not know
the answer to something, then please just say that you don't know.

You are a customer support assistant. Be polite.
Answer accurately. If you don't know, say so.

Now imagine that system prompt is sent on every one of 100,000 daily conversations. Saving ~34 tokens per call is ~3.4 million tokens a day saved on the system prompt alone — and the model behaves identically. That is the everyday economics of compression: small per-call savings, large at volume.

For history, a rolling summary in pseudocode looks like this. Note the cheap model does the summarizing so you don't pay frontier-model prices just to shrink old chatter.

RECENT_TURNS = 6          # keep these verbatim
TOKEN_BUDGET = 3000       # max tokens for the history block

def build_history(turns):
    recent = turns[-RECENT_TURNS:]
    old = turns[:-RECENT_TURNS]

    if not old:
        return render(recent)

    # Summarize everything older than the recent buffer with a
    # cheap, fast model — one summary, not one call per turn.
    summary = cheap_model_summarize(old, max_tokens=400)

    history = f"Earlier conversation summary:\n{summary}\n\n" + render(recent)

    # Safety net: if still over budget, shrink the summary further.
    while count_tokens(history) > TOKEN_BUDGET:
        summary = cheap_model_summarize(summary, max_tokens=250)
        history = f"Earlier conversation summary:\n{summary}\n\n" + render(recent)

    return history

// Compression vs. prompt caching

People often confuse compression with prompt caching, because both lower the cost of long prompts. They are different tools that solve different halves of the problem, and the best systems use both.

The key distinction: compression reduces how many tokens exist; caching reduces how much it costs to reprocess tokens that didn't change. A long system prompt that is identical on every call is a perfect caching candidate — you don't even need to shrink it if caching makes the reprocessing nearly free. But the user's question, the freshly retrieved documents, and the latest chat turn are different every time, so caching can't help there — that variable part is exactly where compression earns its keep.

// Common pitfalls

Compression is easy to overdo. The failure mode is always the same: you shaved off something the model quietly depended on, and now the answers degrade in ways that don't show up until a user hits the exact case you cut.

• Cutting load-bearing detail. A name, a number, a negation ("do not issue refunds after 30 days") can be one token of difference between right and wrong. Aggressive summarization is exactly where these vanish. Compress the filler, guard the facts.
• Compressing the wrong part. Spending effort shrinking a system prompt that caching already makes cheap, while ignoring a giant retrieved-document block that changes every call, optimizes the part that didn't matter.
• Summarizing every turn. Calling an LLM to summarize after every message is slow and can cost more than the tokens you saved. Summarize in batches, only when you cross a token budget.
• No measurement. "It still looked fine on a few questions" is not evaluation. Compression is a quality trade-off — you must measure answer quality before and after on a fixed test set, or you are flying blind.
• Over-trusting automatic tools. Token-dropping tools like LLMLingua can remove a token that turns out to be critical for a specific query. Treat their output as lossy and validate on your real traffic.

// Going deeper

Once the basics click, prompt compression connects to several deeper threads in context engineering and model efficiency.

Retrieval as compression. The cleanest way to send fewer document tokens is to retrieve fewer, better passages in the first place. A good retriever plus a reranker — fetch broadly, then keep only the top few — is itself a form of compression: you never put the irrelevant 95% of a corpus into the prompt. Often, improving retrieval beats compressing whatever retrieval returned.

Structure helps the model and you. Wrapping sections in clear delimiters (see structuring prompts with XML or Markdown) costs a few tokens but makes it safe to compress aggressively elsewhere, because the model can still tell the parts apart. Structure and compression are complementary, not competing.

The reasoning trade-off. Some techniques deliberately add tokens for accuracy — chain-of-thought prompting asks the model to think step by step, which lengthens the output. Compression targets wasteful input tokens, not productive reasoning tokens. Knowing which is which is the whole skill: cut the packaging, keep the parts that do work.

Soft prompts and learned compression. Beyond text-level tricks, research explores compressing context into a handful of trained vectors ("soft prompts" or memory tokens) that stand in for a long passage. These are powerful but model-specific and not yet a mainstream production tool — worth knowing exists, not worth reaching for first.

The durable lesson mirrors the rest of context engineering: the model only has so much room and so much attention, and every token you spend competes with every other. Compression is the discipline of spending those tokens deliberately — measure cost and quality together, compress the bulk, and never touch the few tokens the answer actually hangs on.