What Is HyDE? Hypothetical Document Embeddings Explained

// In plain English

In a normal RAG system you turn the user's question into an embedding — a list of numbers that captures its meaning — and search a vector database for the document chunks whose embeddings sit closest to it. That works, but it has a hidden flaw: a question and an answer rarely look alike. "How long do I have to return a jacket?" and "Refunds on physical items are accepted within 30 days of purchase" share almost no words, and even their meanings sit in slightly different neighborhoods of the embedding space.

HyDE — short for Hypothetical Document Embeddings — fixes this with a move that sounds backwards the first time you hear it. Instead of embedding the question, you first ask a language model to write a fake answer to it, then embed that and search with it. The fake answer is invented and might be wrong, but it doesn't matter: it looks like a real document, so it lands near real documents in the embedding space. You retrieve actual chunks that resemble the answer, not the question.

Think of a librarian searching the stacks. If you tell them only "I have a question about returns," they wander. But if you say "I think the answer is something like you can return physical items within about a month," they instantly walk to the right shelf — even though your guess wasn't exactly right. HyDE hands the search engine a plausible answer to match against instead of a bare question.

// Why it matters

Retrieval is the make-or-break stage of any RAG system. If the correct chunk never lands in the top results, no language model can recover — it answers from whatever junk it was handed. HyDE attacks one specific, common cause of bad retrieval: the question-answer gap.

• Short, vague queries. Real users type three-word questions: "refund window jacket?" There's barely any signal to embed. A generated hypothetical answer is a full, fluent paragraph packed with the vocabulary and phrasing real documents use — far more to match on.
• Vocabulary mismatch. A user asks about "giving back a purchase"; the manual says "return" and "refund." Pure question embedding can miss this. A model writing a hypothetical answer naturally uses the document's own words, closing the gap.
• Zero-shot retrieval with no training data. HyDE's original selling point: it improves retrieval on a brand-new domain without fine-tuning the embedding model on labeled query-document pairs. You get better recall for free, using only a language model you already have.

Who should care? Anyone whose retriever returns plausible-but-wrong chunks on short or oddly-worded questions, especially in a domain where you have no labeled data to fine-tune embeddings. HyDE is a cheap, drop-in upgrade: it changes what you embed, not your vector database, your chunks, or your final-answer prompt. You can bolt it onto an existing pipeline in an afternoon and measure whether recall improves.

// How it works

HyDE inserts exactly one extra step in front of your normal retrieval: a quick language-model call that drafts a hypothetical answer. Everything downstream — the embedding model, the vector search, the final generation — stays the same.

Step by step, here's what changes versus plain RAG:

1. Draft a hypothetical document. Send the question to a language model with a prompt like "Write a short passage that answers this question." The model produces a confident, document-shaped paragraph. It may contain factual errors — that is expected and, surprisingly, usually fine.
2. Embed the draft, not the question. Run that hypothetical answer through your normal embedding model to get a query vector.
3. Search as usual. Ask the vector database for the chunks closest to that vector. Because the draft looks like an answer, the closest chunks tend to be real passages that answer the same question.
4. Generate the real answer. Throw the hypothetical draft away. Feed only the retrieved real chunks plus the original question to the model for the grounded final answer — exactly as in standard RAG.

Here is the whole technique in a few lines of Python. Notice that only the retrieve function differs from a normal pipeline — and within it, only the first three lines are new.

from anthropic import Anthropic

client = Anthropic(api_key="sk-ant-...")

def hypothetical_answer(question):
    # Ask the model to DRAFT a plausible answer. It may be wrong;
    # we only use it as a richer search query, never as the result.
    prompt = (
        "Write a short, factual-sounding passage that could answer "
        f"this question. Do not hedge.\n\nQuestion: {question}"
    )
    msg = client.messages.create(
        model="claude-haiku-4-5",   # small + fast is fine here
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    )
    return msg.content[0].text

def retrieve(question, k=5):
    draft = hypothetical_answer(question)   # <-- the HyDE step
    query_vec = embed([draft])[0]           # embed the DRAFT, not the question
    return vector_db.search(query_vec, k=k) # same search as plain RAG

chunks = retrieve("how long to return a jacket?")
# ...then run normal RAG: stuff `chunks` + the question into the final prompt

A common refinement is to generate several hypothetical answers (say three or four), embed each, and average the vectors. Averaging smooths out the noise: if one draft veers off into a wrong direction, the others pull the query vector back toward the consensus. You can also append the original question's embedding to the mix so a badly hallucinated draft can't drag the search completely off-topic.

// A worked example

Suppose your knowledge base contains this real chunk:

Physical items may be returned within 30 days of the original
purchase date for a full refund. Items must be unused and in
original packaging. Digital goods are non-refundable once downloaded.

Now the user types a terse, casual question. Watch how the two strategies compare on what actually gets embedded and searched:

Notice the draft got the detail slightly wrong — it guessed "about a month" when the policy says 30 days. That error never reaches the user. The draft only steered the search to the right shelf; the final answer is generated from the retrieved real chunk, which states the exact 30-day rule. This is the heart of why HyDE tolerates hallucination: a roughly correct answer is already shaped like the real document, and shape is all the embedding needs.

// HyDE vs query rewriting

HyDE is easy to confuse with query rewriting, its close sibling. Both put a language model in front of retrieval, but they produce different things and fix different problems. Keeping them straight helps you pick the right tool.

The clean way to remember it: *query rewriting improves the question; HyDE replaces the question with a guessed answer.* They're not mutually exclusive — you can rewrite an ambiguous question first, then apply HyDE to the cleaned-up version. And both are different again from a reranker, which acts after retrieval to re-score candidates rather than changing what you search with.

// Common pitfalls and when to skip it

HyDE is a sharp tool, not a default. It has a real failure mode and a real cost, and on some queries it actively hurts. Know when not to reach for it.

• The hypothetical answer is confidently wrong. If the model drafts a plausible-but-misdirected answer, the embedding points at the wrong region and you retrieve confidently irrelevant chunks. This is most dangerous on niche or proprietary topics the model knows nothing about — exactly the domains where you reached for RAG in the first place.
• Exact-match queries. For an error code, a product SKU, a person's name, or a precise quote, you want the literal string, not a paraphrased answer. HyDE can blur these. Pair it with keyword search — see hybrid search and BM25 — so exact matches still surface.
• Added latency and cost. Every query now needs an extra LLM call before retrieval even starts. Generating several drafts multiplies that. On a high-traffic system this is a measurable tax — use a small, fast model for the draft.
• No evaluation. Whether HyDE helps depends entirely on your corpus and your queries. "It felt better" is not data. Measure retrieval recall with and without HyDE on a real question set before shipping it.

// Going deeper

Once the basic idea clicks, a few nuances separate a demo from a system you'd trust in production.

HyDE shines exactly where embedding models are weakest: zero-shot, out-of-domain retrieval. The original paper's whole point was matching or beating fine-tuned retrievers without any labeled query-document pairs. If you've already fine-tuned a strong domain embedding model on your own data, the question-answer gap is smaller and HyDE's benefit shrinks — sometimes to nothing. The honest takeaway: HyDE is most valuable when you can't train a good retriever, and least valuable when you already have one.

Tune the draft to your documents. A hypothetical answer should resemble the form of your chunks. If your knowledge base is terse API reference, prompting the model to write a flowery essay creates a mismatch. Steer the draft's style and length toward what actually lives in your index — that alignment, more than the draft's factual accuracy, drives retrieval quality.

It composes with the rest of the retrieval stack. HyDE changes only the query vector, so it stacks cleanly with hybrid search, fusion, and a cross-encoder reranker. A strong modern pipeline often looks like: rewrite or HyDE the query → run dense + BM25 retrieval in parallel → fuse the lists → rerank the top candidates → generate. HyDE is one knob among several, not a replacement for the others.

The deeper lesson is about asymmetry. Most retrieval pain comes from comparing two things of different shapes — a short question against long answers. HyDE's trick is to make both sides the same shape (answer-vs-answer) before comparing. That framing recurs across retrieval research: whenever query and document look too different, ask whether you can transform one to resemble the other. The most reliable way to learn whether it helps you is still the unglamorous one — build an evaluation set and measure recall, because on retrieval, intuition is frequently wrong.