What Is Multi-Hop RAG? Answering Questions That Need Several Steps

// In plain English

Plain RAG does one lookup: it takes your question, finds the most relevant chunks, and hands them to the model. That works beautifully when the answer sits in a single passage. But some questions need you to find one fact before you even know what to look up next.

Take a question like: "Who is the CEO of the company that makes the GPUs our training cluster runs on?" You can't answer that in one search. First you have to learn which GPUs the cluster uses, then learn which company makes them, then look up who runs that company. Three lookups, and each one depends on the answer to the one before it. A single retrieval has no idea the second and third questions even exist.

Multi-hop RAG (also called iterative or multi-step retrieval) handles exactly this. Instead of retrieve once, then answer, it loops: retrieve, reason about what's still missing, retrieve again — repeating until it has every fact the answer needs. Each pass is a "hop."

Picture a detective, not a librarian. A librarian fetches the one book you asked for and is done. A detective pulls a thread, reads it, realizes it points to a new suspect, goes back to the files for that person, and keeps going until the chain of clues is complete. Multi-hop RAG turns your retriever into the detective.

// Why it matters

Once a RAG demo leaves the "what's our refund window?" stage and meets real user questions, single-shot retrieval starts failing in a way that's hard to debug: the model gives a confident, wrong answer, and the retrieved chunks look relevant — they just weren't the chunks that mattered.

The root cause is that the first search only sees the original wording of the question. If a key entity in the answer is never named in the question, its chunk has nothing to match against, so it's never retrieved. Multi-hop RAG fixes this by letting later searches use facts discovered in earlier ones.

The kinds of questions that need it

• Bridge questions — the answer hinges on an intermediate entity the question never names. "What's the capital of the country that won the 2018 World Cup?" You must find the winner (France) before you can look up its capital.
• Comparison questions — you need facts about two or more things, then a comparison. "Which of our three cloud providers has the strictest data-retention policy?" requires retrieving each policy separately.
• Aggregation questions — the answer combines evidence scattered across many documents. "How many of last quarter's incidents were caused by the same root component?"
• Temporal / causal chains — "What changed in the deploy that came right before the outage?" needs you to find the outage time first, then the deploy before it.

If your users ask anything beyond fact-lookup — questions with the words "that," "which," "compared to," or "because" doing real work — single-shot RAG will quietly underperform, and adding more chunks to the prompt won't save it. The missing fact was never retrieved in the first place; you can't rerank your way to a chunk you never fetched.

// How it works

Multi-hop RAG wraps a normal retrieve-and-generate step inside a controller loop. After each retrieval, the system asks itself a single question: do I now have enough to answer? If yes, it generates the final answer. If no, it forms a new, more specific query from what it just learned and searches again.

Three pieces do the work, and the loop is just glue between them:

• A controller decides whether to keep going and what to search for next. Usually this is the LLM itself, prompted to inspect what it has and name the next sub-question.
• A retriever runs each hop's search against your vector store or index — the same retriever a one-shot system uses, just called repeatedly.
• A working memory (sometimes called the scratchpad or accumulated context) carries the facts gathered so far from one hop into the next, so each query can build on the last.

Decompose, or follow the thread

There are two common shapes for the loop, and real systems blend them:

Up-front decomposition. Before any search, the LLM breaks the question into ordered sub-questions, then answers them in sequence, feeding each answer into the next query. This works when the structure is clear from the start (comparisons, aggregations).

Follow-the-thread (interleaved). The system searches, reads what came back, and only then decides the next query based on what it found — because you often can't know the second question until you've answered the first. This is the more powerful and more general pattern; the influential self-ask and IRCoT approaches both interleave retrieval with the model's own chain of reasoning.

A sketch of the loop in code

def multi_hop_answer(question, retriever, llm, max_hops=4):
    facts = []                      # working memory across hops
    query = question

    for hop in range(max_hops):
        chunks = retriever.search(query, k=4)   # one retrieval per hop
        facts.extend(chunks)

        # Controller: is the gathered evidence enough yet?
        decision = llm.ask(
            f"Question: {question}\n"
            f"Evidence so far:\n{format(facts)}\n\n"
            "If you can now answer, reply DONE. "
            "Otherwise reply with the single next thing to look up."
        )

        if decision.strip().startswith("DONE"):
            break
        query = decision               # next hop searches for the gap

    return llm.ask(
        f"Answer using ONLY this evidence:\n{format(facts)}\n\n"
        f"Question: {question}"
    )

That's the entire idea: a for loop, a retriever called inside it, and an LLM acting as the controller that decides when to stop. Everything sophisticated — better stopping rules, deduplicating evidence, reranking across hops — is a refinement of this skeleton.

// A worked example, hop by hop

Let's trace a real bridge question through the loop. The knowledge base is a company wiki, and the question is: "Which team owns the service that caused the longest outage last quarter?" Notice the answer (a team) is never mentioned in the question — you have to discover the outage and the service first.

Each query is built from the previous hop's answer, which is why no single search could have done it. A one-shot retriever searching "which team owns the service that caused the longest outage" would match chunks about teams, services, and outages all at once — a blurry mix — and very likely never surface the specific INC-204 → Payments API → Billing chain. The loop walks the chain one verified link at a time, and the final answer can cite all three source chunks, which makes it easy to add citations.

// Multi-hop RAG vs agentic RAG

These two terms get used almost interchangeably, and they overlap heavily — but they answer different questions. *Multi-hop RAG describes the retrieval loop: search, reason, search again.* *Agentic RAG describes the broader framing: give the model retrieval as one tool among several and let it drive.* Multi-hop is, in effect, the simplest agentic RAG, where the only tool is search.

Practically: if your system only searches a knowledge base, just over and over, you're doing multi-hop RAG. The moment it can also call a calculator, hit a live API, or run code between searches, it's an agent and the multi-hop retrieval is one behavior inside it. Build the multi-hop loop first — it's the part that makes complex question-answering work — and reach for the full agentic setup only when one tool genuinely isn't enough.

// Common pitfalls

Multi-hop RAG adds power and, with it, new ways to fail. Almost all of them come from the loop itself — the part a single-shot system doesn't have.

• Error compounding. A wrong fact in hop 1 poisons every hop after it. If the controller misreads INC-204's service, hops 2 and 3 confidently chase the wrong team. Early mistakes are the most expensive — verify intermediate facts before building on them.
• Runaway loops. A controller that never feels "sure enough" keeps retrieving. Always cap hops and add an explicit stop condition; treat hitting the cap as a failure signal, not a quiet success.
• Latency and cost stack up. Each hop is at least one retrieval plus one LLM call, run in sequence. A 4-hop answer can be 4× slower and pricier than single-shot. Reserve multi-hop for questions that truly need it — route simple lookups to plain RAG.
• Context bloat. Accumulating every chunk from every hop floods the final prompt with noise. Keep a tight working memory: store the extracted fact, not the whole raw chunk, and dedupe across hops.
• Decomposing a question that didn't need it. Forcing a simple lookup through a multi-hop loop wastes calls and can introduce errors. The controller should be allowed to answer in one hop when one hop is enough.

// Going deeper

The sketch above is the naive multi-hop loop. The research and production work beyond it is all about making the loop stop at the right time, trust the right facts, and not cost a fortune.

Better stopping and verification. Instead of trusting the controller's gut, you can score whether the gathered evidence actually entails an answer, or have a separate verifier check each intermediate fact before the next hop builds on it. Self-RAG-style approaches train the model to emit reflection tokens — judging relevance and support as it goes — which gives the loop a more principled stop signal than a raw yes/no.

Query planning vs interleaving, revisited. Decomposing everything up front is fast and parallelizable but brittle when the structure isn't knowable in advance; interleaving is robust but strictly sequential and slower. Mature systems detect which mode a question needs — run independent sub-questions in parallel (a comparison of three policies), and reserve the sequential thread-following loop for genuine bridge questions.

Where structure helps more than loops. When questions routinely span many documents through named relationships, a knowledge-graph approach (GraphRAG) can answer in one structured traversal what would otherwise take many vector hops. And hybrid retrieval matters more in multi-hop, because intermediate facts are often exact strings — an incident ID, a service name, an SKU — that keyword search nails and pure semantic search can miss.

Knowing when not to. Multi-hop is a cost you pay per question. If your traffic is overwhelmingly single-fact lookups, a router that sends easy questions to plain RAG and only the hard ones into the loop beats running everything multi-hop — see when not to use RAG for the broader cost calculus. The durable lesson: multi-hop RAG buys you reasoning over several facts, but it inherits every weakness of single-shot retrieval and adds compounding errors and latency on top, so spend it only where a single lookup genuinely can't reach the answer.