What Is Cohere Rerank? Managed Reranking API

// In plain English

In a RAG system, the first step is search: you ask a vector database for the passages that look most relevant to a question. That first pass is fast and cheap, but it is also a bit blurry — it sorts results by rough similarity, so the truly best answer is often sitting at position 7 instead of position 1. Cohere Rerank is a hosted API that takes those rough results and reorders them by how well each one actually answers the query.

Think of hiring. A recruiter does a keyword screen and pulls 50 résumés that broadly match the job — that is your first-stage retrieval. Then a senior engineer reads each one next to the job description and ranks the top 5 they would actually interview. Cohere Rerank is that senior engineer: it reads the query and each candidate document together, side by side, and scores how good a match they really are. You send it a query plus a list of documents; it sends back the same documents with a relevance score and a new order.

// Why it matters

First-stage retrieval has to be fast because it searches your whole corpus — sometimes millions of chunks. To stay fast, it compares pre-computed embeddings: the query becomes one vector, every document is already a vector, and the system just measures distance. That is a bi-encoder approach, and it is great at recall (pulling back a pool that probably contains the answer) but only so-so at precision (putting the single best chunk on top).

That gap is exactly what hurts a RAG answer. You usually only paste the top 3–5 chunks into the prompt, because context is expensive and models get "lost in the middle" of long prompts. If the genuinely best passage is ranked 8th, it never reaches the model, and the answer suffers — no matter how capable the model is. A reranker fixes the ordering of that small candidate pool so the best material lands in the top few slots you actually use.

• Higher precision where it counts. Retrieve a broad pool of 50–100 candidates cheaply, then let the reranker pick the true top 5. You feed the model better context without widening the prompt.
• No model to host. A strong reranker is a cross-encoder model that needs a GPU and serving infrastructure. Cohere Rerank is an HTTP call — you add one step to your pipeline and Cohere runs the model.
• Drop-in upgrade. Reranking sits after whatever search you already have (vector, keyword, or hybrid). You do not have to re-embed your corpus or change your database to add it.
• Multilingual and long-document handling. The hosted models handle many languages and longer passages, which is otherwise fiddly to get right when self-hosting.

The trade-off is that you are now sending data to an external API and paying per search. For many teams that is a fine deal; for some it is a dealbreaker. We cover that decision below.

// How it works

Reranking is a two-stage retrieval pattern. Stage one is your existing fast search, which casts a wide net. Stage two is the reranker, which reads the query and each candidate together and produces a precise relevance score. You then keep the top few and pass them to the LLM.

Why the second stage is more accurate

First-stage search uses a bi-encoder: the query and the documents are turned into vectors separately, so the model never sees them at the same time. It is fast because document vectors are computed once, in advance — but the model can't reason about how a specific query relates to a specific document. A reranker uses a cross-encoder: it feeds the query and one document into the model together and outputs a single relevance score. Because the two texts attend to each other, it catches nuance a vector distance misses — negation, specific entities, who-did-what-to-whom. The price is that it must run once per query-document pair, which is why you only use it on the small candidate pool, not the whole corpus. The deeper contrast lives in cross-encoder vs bi-encoder.

What the API call looks like

The interface is deliberately small. You pass a query, a list of documents, and how many results you want back (top_n). The response is the documents reordered, each with a relevance_score and its original index so you can map back to your data. Your first-stage search and your database stay exactly as they are — reranking just slots in between.

import cohere

co = cohere.ClientV2(api_key="...")  # your Cohere API key

query = "What is the refund window for physical items?"

# These came from your first-stage search (vector / keyword / hybrid).
# Usually 50-100 rough candidates, not the whole corpus.
candidates = [
    "Digital goods are non-refundable once downloaded.",
    "Refunds on physical items are accepted within 30 days of purchase.",
    "Support hours are 9am to 6pm Eastern, Monday to Friday.",
    # ... dozens more ...
]

resp = co.rerank(
    model="rerank-v3.5",     # use the current model id from the docs
    query=query,
    documents=candidates,
    top_n=3,                 # keep only the best few for the prompt
)

# results come back sorted best-first, with a score and original index
for r in resp.results:
    print(r.index, round(r.relevance_score, 3), candidates[r.index])

# Take the reranked top-n and stuff them into your RAG prompt as context.

// A worked example: rescuing the buried answer

Suppose a user asks: "Can I return a physical product after three weeks?" Your vector search returns 50 chunks. Embedding similarity is dominated by surface words, so chunks that mention "product", "return", and "weeks" float up — including a chunk about digital returns and one about shipping times. The chunk that actually answers the question ("physical items: 30-day return window") happens to land 6th, because it phrases the rule differently.

If you only keep the top 3 for your prompt, the right answer never reaches the model, and it may wrongly say "digital products can't be returned" by pattern-matching the wrong chunk. Now add a rerank stage over those 50 candidates.

The reranker read the full query against each chunk and understood that "after three weeks" + "physical product" maps onto the 30-day physical-returns rule, not the digital or shipping passages. It promoted the right chunk to position 1. Same search, same database, same documents — one extra API call moved the correct answer into the slot the model actually reads.

// Cohere Rerank vs self-hosting a reranker

The reranking pattern is the same either way. The choice is whether someone else runs the model (Cohere Rerank) or you do (an open cross-encoder you host, like the BGE reranker family). Pick based on data sensitivity, cost shape, and how much infrastructure you want to own.

A common path: start with the managed API to prove that reranking helps your numbers, then revisit self-hosting only if cost at scale or a data-residency rule forces the move. Do not self-host on day one for a problem you have not yet measured.

// Common pitfalls

• Reranking too few candidates. A reranker can only reorder what you give it. If first-stage search returns 5 chunks and the answer wasn't in those 5, reranking can't conjure it. Retrieve a wide pool (e.g. 50–100) so the right chunk is present to be promoted.
• Reranking the whole corpus. The opposite mistake. A cross-encoder runs once per query-document pair, so reranking 100,000 chunks per query is slow and expensive. Always rerank a small candidate pool, never the full index.
• Forgetting the latency cost. Adding a network round trip plus model inference adds real milliseconds to every query. Usually worth it, but measure it and keep the candidate pool sized sensibly.
• Hard-coding a score threshold. Relevance scores are meaningful for ordering, but a fixed cutoff that works on one dataset can silently drop good results on another. Tune any threshold on your own evaluation set.
• Skipping evaluation. "It feels better" is not proof. Measure whether reranking actually lifts retrieval quality (did the right chunk reach the top-k?) before and after — otherwise you are paying for a step you can't justify.

// Going deeper

Once the basic two-stage setup is working, a few refinements separate a decent reranking step from a great one.

Tune the candidate-pool size. The width of your first-stage pool (often called the retrieve depth) is the single biggest lever. Too narrow and the answer never enters the pool; too wide and you pay more latency and cost for diminishing returns. Sweep a few values against an evaluation set and pick the smallest pool that keeps recall high.

Pair it with strong first-stage search. Reranking does not excuse weak retrieval — it sharpens a good pool, it does not rescue a bad one. The strongest setups feed the reranker a hybrid search pool that blends semantic vectors with keyword scoring (BM25), so exact matches like error codes and product names are present and meaning-based matches are too. Improving query rewriting before retrieval helps the reranker even more.

Combine reranking with metadata filtering. Reranking decides order by content; metadata filtering decides eligibility by attributes (tenant, language, date, permissions). Filter first to remove documents a user should never see, then rerank what remains. Order matters: never let a reranker promote a chunk the user wasn't allowed to retrieve.

Mind versions without depending on them. Hosted rerank models are updated over time, and the exact model id you pass will change across generations. Read the current model name and limits from the official docs rather than pinning to whatever you saw in a tutorial, and re-run your evaluation when you upgrade — a newer model can shift scores and reorder edge cases.

The durable takeaway: reranking is the cheapest large win available in most RAG pipelines, because it improves the order of context the model sees without touching your data, your embeddings, or your database. Cohere Rerank is simply the managed way to add that win in a single API call — start there, measure the lift, and only graduate to a self-hosted reranker when cost or privacy actually demands it.