What Is the BGE Reranker? Open Cross-Encoder

// In plain English

When a RAG system searches your documents, the first pass is fast but rough. It turns the question into one vector, compares it against pre-computed document vectors, and returns whatever sits closest. That speed comes at a cost: the question and each document are read separately, so the search only knows roughly what each one is about, not how well they actually fit together.

The BGE reranker is the careful second reader. It is an open-source model from BAAI (the Beijing Academy of Artificial Intelligence) that takes the question and one document at the same time, reads them together, and gives back a single relevance score. You run it over the handful of candidates the fast search returned, then keep only the best ones. The Chinese acronym BGE stands for BAAI General Embedding, the family this reranker ships alongside.

Think of hiring. The fast search is the recruiter who skims 500 resumes by keyword in a minute and forwards 20 that look plausible. The reranker is the hiring manager who actually sits down with each of those 20 next to the job description and ranks them properly. The recruiter trades depth for speed across a huge pile; the manager trades speed for depth across a short list. You need both, in that order.

// Why it matters

In RAG, the model can only answer from the chunks you put in its prompt. If the truly relevant passage never makes it into that short list, no amount of clever prompting saves the answer. Reranking is the cheapest, highest-leverage way to make sure the right chunk lands near the top — and the BGE reranker is the default way to do it without paying a per-call API.

What problem it solves

• Vector search is imprecise. Bi-encoder retrieval compresses a whole document into one vector. Two passages can look close in that compressed space yet answer different questions. A cross-encoder reads the full text of both sides, so it catches mismatches the vector distance missed.
• Top-k is often wrong at the top. The genuinely best passage frequently sits at rank 5 or rank 15, not rank 1. Reranking re-sorts the candidates so the best one moves up where the model will actually read it.
• You can retrieve loosely, then tighten. Because a reranker fixes ordering, you can let the first stage cast a wide net (high recall, some junk) and trust the reranker to clean it up. That two-stage split is the backbone of most production RAG.

Why the BGE one specifically

There are hosted reranking APIs you can call instead. The BGE reranker matters because it is open-weight — you download it and run it on your own hardware. That gives you three things a managed API can't always promise: no per-query fee no matter how much traffic you push through it, data that never leaves your servers (important for private or regulated documents), and the freedom to fine-tune it on your own domain. It is also multilingual, so it works across many languages rather than English alone. For teams that want reranking without a vendor in the loop, it has become the default starting point.

// How it works

A reranker is a second stage. The fast retriever does the heavy lifting of narrowing millions of chunks down to a few dozen candidates; the reranker then does the slow, accurate scoring on just that short list. Putting the expensive model last is what makes the whole thing affordable.

Inside the cross-encoder

For every candidate, the reranker joins the query and the document into a single input — roughly [query] [separator] [document] — and feeds the whole thing through a transformer. Because both texts share the same pass, the model's attention layers can compare each word of the question against each word of the document directly. The output is one number: a relevance score for this specific pair. You repeat that for each candidate, sort by score, and the new order is your reranked list.

Notice the catch: the reranker must run once for every candidate, and nothing can be pre-computed because the score depends on the query. That is exactly why it lives in stage two over a short list, never as a first-pass search over your whole corpus. The numbers it returns are relevance scores for ranking, not calibrated probabilities — only their order relative to each other is meaningful.

from FlagEmbedding import FlagReranker

# Load the open-weight model once at startup (downloads from Hugging Face).
reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

query = "How long do I have to return a physical product?"

# These came from your fast first-stage retriever (vector or BM25).
candidates = [
    "Digital goods are non-refundable once they have been downloaded.",
    "Refunds on physical items are accepted within 30 days of purchase.",
    "Our support hours are 9am to 6pm Eastern, Monday to Friday.",
]

# Score every (query, document) pair, then sort best-first.
pairs = [[query, doc] for doc in candidates]
scores = reranker.compute_score(pairs)

ranked = sorted(zip(scores, candidates), reverse=True)
for score, doc in ranked:
    print(round(score, 3), doc)

# The refund-window passage now ranks first, even though plain vector
# search may have ranked the 'non-refundable' line just as close.

// Bi-encoder vs cross-encoder: the core tradeoff

Understanding the BGE reranker really means understanding why it is not used for first-stage search. The split comes down to one design choice: do you encode the query and document apart, or together?

Because document vectors can be computed ahead of time, a bi-encoder searches a huge index almost instantly — that is what makes vector search scale. But that same shortcut is why it is approximate: it never sees the query and document side by side. The cross-encoder gives up the pre-computation to gain that joint view. The two are partners, not rivals: the bi-encoder makes search possible, the cross-encoder makes the top of the list correct. This is covered in depth in cross-encoder vs bi-encoder.

// Where it fits in a real pipeline

The reranker rarely works alone. The standard modern recipe is hybrid search → fuse → rerank, and the BGE reranker is the final tightening step.

• Cast a wide net. Run two first-stage searches in parallel: dense vector search for meaning and BM25 keyword search for exact terms like error codes and product names. This is hybrid search.
• Merge the two lists. Combine the dense and keyword results into one candidate pool, often with reciprocal rank fusion, which blends rankings without needing the scores to be on the same scale.
• Rerank the pool. Feed the merged candidates (commonly 20 to 100 of them) through the BGE reranker, which re-scores every (query, document) pair and re-sorts them by true relevance.
• Keep only the best. Pass the top few — often 3 to 8 — into the LLM prompt. Fewer, better chunks usually beat more, noisier ones.

For the general concept behind this stage — including hosted alternatives and how to measure its impact — see what is reranking.

// Going deeper

Once the basic retrieve-then-rerank loop is working, a few finer points separate a demo from a dependable system.

Latency is the real constraint. A cross-encoder runs the model once per candidate, so reranking 100 passages means 100 forward passes. On a CPU that can dominate your response time; reranking is much happier on a GPU, and you can keep the batch small by reranking 20–50 candidates rather than hundreds. The BGE family also ships in different sizes — smaller variants are faster and lighter, larger ones score a touch more accurately — so model size becomes a speed-versus-quality dial you choose deliberately.

Long documents need care. A cross-encoder has a maximum input length, and the query plus document must fit inside it together. If your chunks are long, the tail can get truncated and silently ignored. Keep chunks reasonably sized, or make sure the part most likely to answer the query sits near the start. The multilingual bge-reranker-v2-m3 variant is the common default precisely because it handles longer inputs and many languages well.

Self-host vs managed API. A hosted rerank API removes all the GPU and deployment work, and for low traffic it is often the pragmatic choice. The BGE reranker wins when traffic is high enough that per-query fees add up, when documents can't leave your environment for compliance reasons, or when you want to fine-tune the reranker on your own labeled query-document pairs to lift domain accuracy. It is genuinely the same job, just owned by you instead of a vendor.

Know the alternatives. A cross-encoder is the accurate-but-slow end of the spectrum. At the other end, ColBERT-style late interaction keeps token-level embeddings to get some of the cross-encoder's precision while staying closer to bi-encoder speed. And reranking is only as good as the candidates handed to it — if the right passage never reaches the short list, no reranker can promote it, so first-stage recall and chunking still set your ceiling. Rerank to fix ordering, not to rescue a search that missed.