What Is Query Rewriting in RAG?

// In plain English

In a RAG system, the user's question is what you search with. You embed it, hand it to the retriever, and it pulls back the documents that look most similar. So the whole pipeline rests on one quiet assumption: that the way a person phrases a question lines up with the way the answer is written in your documents.

That assumption breaks constantly. People type "why is it broken again" when the manual says "intermittent connection loss after firmware update." They ask three things in one breath. They use a nickname for a product the docs call by its official name. The answer is sitting right there in your corpus — but the question, as written, doesn't match it, so retrieval comes back with junk.

Query rewriting is the fix: before you search, you reshape the user's question into something that retrieves better. You might clean up vague phrasing, add synonyms, split a compound question into separate searches, or zoom out to a broader version of the question. Then you search with the improved version. The umbrella term for this family of tricks is query transformation.

// Why it matters

Here is the uncomfortable truth about RAG: most failures are retrieval failures, and most retrieval failures start with the query. If the right chunk never enters the prompt, no model on earth can answer from it. You can swap in a bigger LLM, add a reranker, tune your chunk size — and still get bad answers, because the search was doomed before it began by the shape of the question.

Query rewriting earns its keep on the kinds of questions real users actually ask:

• Vague or conversational. "How do I fix this?" or "is it down?" carry almost no searchable signal on their own — especially with no conversation context attached.
• Vocabulary mismatch. The user says "sign-in," the docs say "authentication." Pure semantic search bridges some of this gap, but not exact terms like error codes, SKUs, or rare proper nouns.
• Compound questions. "What's the refund window, and does it apply to digital goods?" is two questions. One embedding tries to be similar to both topics at once and ends up firmly matching neither.
• Too specific to match the framing. "Was Einstein born before or after the telephone was invented?" needs two separate facts that no single chunk states together; a literal search for the whole sentence finds nothing useful.

The payoff is leverage in the cheapest possible place. Rewriting happens before embedding and search, so a small improvement to the query cascades into better candidates, which means a better rerank, which means a better final answer. It's the one stage where a little extra thinking up front pays off through the entire rest of the pipeline. The cost is latency — you've added an LLM call (or several) before retrieval — and that tradeoff is the heart of choosing which technique to use.

// How it works

The mechanism is simple: you insert a transformation step between the raw question and the retriever. In naive RAG, the user's text goes straight to embedding. With query rewriting, an LLM (or a rule) reshapes it first, and that output is what you search with.

"Rewrite step" is a family, not one thing. Four techniques cover most of what you'll meet, and they differ in how many queries they produce and how aggressively they change the meaning.

1. Rewriting for clarity

The lightest move: clean the question up without changing what it asks. Resolve pronouns ("is it down?" → "is the payments API down?"), fold in conversation history, strip filler. This matters most in chat, where a follow-up like "and the mobile one?" is meaningless to a retriever until you rewrite it into a standalone, self-contained question.

2. Query expansion

Add alternative phrasings and synonyms so the search net is wider. "How do I reset my password" might expand to also cover "recover account," "forgot login," "change credentials." You can do this with the LLM or with a classic thesaurus/keyword list. Expansion especially helps keyword search like BM25, which is blind to synonyms by design.

3. Multi-query (sub-question splitting)

Take one question and produce several queries, run all of them, and merge the results. This is the answer to compound questions: split "refund window and digital goods?" into two searches, each clean and focused. You then dedupe and combine the hits — often with reciprocal rank fusion, which merges multiple ranked lists fairly. It's also the natural partner for hybrid search, where you already have several result lists to fuse.

4. Step-back prompting

Sometimes the question is too specific to retrieve well. Step-back prompting asks the LLM to generate a broader, more general version first — "what's the refund policy for SKU-4471 bought on a Tuesday?" steps back to "what is the refund policy?" — retrieves the general principle, and lets the generator apply it to the specific case. You retrieve the foundational chunk that actually exists in your docs instead of hunting for an exact match that doesn't.

In code, the transformation is just a focused LLM call whose output you feed to the retriever instead of the raw text:

from anthropic import Anthropic

client = Anthropic(api_key="sk-ant-...")

def rewrite_to_subqueries(question: str) -> list[str]:
    prompt = (
        "Split the user's question into 1-3 standalone search queries.\n"
        "Resolve any pronouns. Output one query per line, nothing else.\n\n"
        f"Question: {question}"
    )
    msg = client.messages.create(
        model="claude-haiku-4-5",          # small + fast: this runs before every search
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    )
    lines = msg.content[0].text.strip().splitlines()
    return [q.strip("- ").strip() for q in lines if q.strip()]

# Retrieve for each sub-query, then merge (e.g. reciprocal rank fusion).
queries = rewrite_to_subqueries("What's the refund window, and does it cover digital goods?")
hits = [chunk for q in queries for chunk in retrieve(q, k=5)]
context = dedupe_and_fuse(hits)

// Which technique, and what it costs

These techniques aren't ranked best-to-worst — they target different failure modes. Match the technique to the symptom you're actually seeing in bad retrievals.

A practical decision rule: look at your failed queries, not your successful ones. If failures are short and ambiguous, start with clarity rewriting. If they fail because the docs use different words, expand. If a single question is really several, go multi-query. If detailed questions retrieve nothing because no chunk is that specific, step back.

And don't reach for any of this on day one. The honest baseline is: ship naive RAG, measure where it fails, and add the one transformation that fixes your most common failure. Every technique here adds latency and complexity, and most pipelines need far less of it than they think.

// Query rewriting vs HyDE

A close cousin you'll meet is HyDE (Hypothetical Document Embeddings). It's worth contrasting because the two are easy to confuse, yet they transform the query in opposite directions.

The shared idea is the same insight: the raw user question is often a poor search key, so reshape it into something that lives closer to your documents in embedding space. Query rewriting reshapes it into a clearer question. HyDE reshapes it into a hypothetical answer — because answers tend to be phrased like the documents you're trying to retrieve. They're siblings, not rivals; both are forms of query transformation, and you can even combine them.

// Going deeper

Once the basics click, a few sharper edges are worth knowing.

It's a latency and cost decision, not just a quality one. Every rewrite is an LLM call on the critical path, before the user sees anything. Multi-query multiplies your retrieval load too. In a latency-sensitive product, the right answer is sometimes "don't rewrite" — or cache rewrites for common questions, or only trigger rewriting when a cheap confidence check says the raw retrieval looks weak.

Rewriting can also hurt. An over-eager LLM may "correct" a query into something subtly different — drop the one rare term that mattered, or generalize away a crucial constraint. A common safety net is to keep the original query in the mix: search with both the rewrite and the raw question, then fuse. You get the upside of rewriting without betting everything on it being right.

Conversational rewriting is its own subfield. In multi-turn chat, turning a context-dependent follow-up into a standalone query ("query contextualization" or "condensing") is often the single highest-impact rewrite you can add — without it, every follow-up retrieves on a fragment that means nothing on its own.

Where it fits with everything else. Query transformation is the front of retrieval; reranking and fusion are the back. A strong production stack often does all three: rewrite or fan out the query, retrieve broadly with hybrid search, then rerank the merged candidates narrowly. They compound — better queries give the reranker better raw material to work with.

The agentic endpoint. Push this idea far enough and the model stops doing a single fixed rewrite. Instead it decides whether to search, what to search for, reads the results, and reformulates the query itself if they look thin — searching again in a loop. At that point query rewriting has dissolved into the agent's own reasoning, but the founding insight still holds: the query you search with matters more than almost anything else in the pipeline, so it deserves real effort before you ever hit the index.