Reranking
A reranker re-scores a small candidate set with a more expensive model so the few chunks that reach the LLM are the right ones — it fixes precision, not recall.
Two-stage retrieval
A first-stage retriever must be cheap because it touches the whole corpus: it scores millions of chunks in milliseconds and returns, say, 20–50 candidates with decent recall and mediocre ordering. A reranker then scores only those candidates with a model that is far more accurate per pair but far too slow to run against everything, and returns the best 5.
The division of labour is deliberate. Stage one optimises recall — is the right chunk somewhere in the top 50? Stage two optimises precision — are the 5 chunks we actually send the best of those 50? Each stage can be measured and tuned on its own (RAG Evaluation).
Bi-encoders vs cross-encoders
A bi-encoder — the embedding model — encodes the query and each chunk *independently* into vectors and compares them afterwards. That independence is what makes it fast: chunk vectors are computed once at ingestion and the query only needs one forward pass plus a nearest-neighbour search. The price is that the model never sees query and chunk together, so it cannot notice that a chunk mentions all the right words in the wrong relationship ("refunds are *not* available for annual plans").
A cross-encoder feeds the query and the chunk into one transformer as a single sequence and outputs a relevance score. Every token of the query attends to every token of the chunk, which is why it is much better at fine distinctions — negation, which entity a number belongs to, whether the chunk answers the question or merely discusses the topic. The price is that nothing can be precomputed: each (query, chunk) pair is a full forward pass, so scoring 1M chunks per query is impossible and scoring 20 is routine.
LLM-based rerankers are the same idea with a bigger model: prompt an LLM with the query and the candidates and ask it to rank or score them. More accurate again, slower and more expensive again, and non-deterministic unless you constrain the output.
- Bi-encoder: precompute chunks, compare vectors,
O(1)model calls per query. Retrieval. - Cross-encoder: one forward pass per pair, sees both texts jointly. Reranking.
- LLM reranker: highest quality, highest cost; useful for the final 10 → 3 cut.
- The same chunk can rank #17 by cosine and #1 by cross-encoder — that gap is the reranker's value.
Cost and latency
Reranking 20 candidates with a small cross-encoder adds roughly 50–200 ms on a GPU or a managed API, and a few cents per thousand queries. Reranking 100 candidates costs 5× that. An LLM reranker over 20 candidates of 400 tokens each is an 8k-token prompt — comparable to the answer generation itself. Pick the candidate count from data: measure recall@N of the first stage and choose the smallest N that captures the relevant chunk in, say, 95% of golden-set queries.
Latency matters more for interactive use than for batch. In a chat UI, a 150 ms reranker is invisible next to a 3 s generation; in an autocomplete-style feature it is the whole budget. Cache reranker scores keyed on (query, chunk_id) for repeated queries, and skip reranking when the first-stage top result is far ahead of the second.
1def retrieve_and_rerank(question: str, *, first_k: int = 30, final_k: int = 5) -> list[Chunk]:2 candidates = hybrid_search(question, k=first_k) # cheap, recall-oriented3 pairs = [(question, c.text) for c in candidates]4 scores = cross_encoder.predict(pairs) # one forward pass per pair5 ranked = sorted(zip(candidates, scores), key=lambda p: p[1], reverse=True)6 top = [c for c, s in ranked[:final_k] if s > 0.2] # threshold learned from evals7 log.info("rerank", moved=[(c.id, i, candidates.index(c)) for i, c in enumerate(top)])8 return topWhen reranking fixes the problem — and when it cannot
Reranking helps when the diagnosis is "the right chunk is retrieved but not in the top 5". Symptoms: recall@20 is high but recall@5 is low; the LLM gets distracted by topical-but-wrong neighbours; near-duplicate chunks crowd the top; queries with negation or specific entities go wrong. Adding a reranker typically lifts precision@5 substantially and lets you send fewer chunks, which also improves generation.
Reranking cannot help when the right chunk is not in the candidate set at all — that is a recall problem in chunking, embeddings, filtering, or hybrid retrieval. Check recall@N of the first stage before adding a reranker; a cross-encoder reordering 20 wrong chunks produces a beautifully ranked list of wrong chunks.
A reranker is also the natural place to fold in signals the first stage ignores: recency boosts, document authority, a penalty for chunks from the same document already selected (diversity), or a business rule preferring official docs over forum posts.
Key points
- Retrieve wide with a cheap model, rerank narrow with an expensive one.
- Bi-encoders precompute and compare; cross-encoders read query and chunk together and are far more precise.
- Reranking improves precision@k, not recall — verify the right chunk is in the candidates first.
- Choose the candidate count N from first-stage recall@N on a golden set.
- Budget 50–200 ms and a small per-query cost for 20–30 candidates; cache scores for repeated queries.
- Use the rerank step to add recency, diversity, and authority signals.
When to use — and when not to
- recall@20 is good but precision@5 is poor.
- Queries involve negation, specific entities, or fine distinctions between similar chunks.
- You want to send fewer, cleaner chunks to reduce generation errors and cost.
- Fusing dense and sparse candidates and you need a single accurate ordering.
- The right chunk is not in the candidate set — fix retrieval first.
- Latency budget is tens of milliseconds total.
- First-stage precision is already at target on your evals.
- Corpus is tiny enough to send everything to the LLM.
Failure modes
- Reranker added to fix a recall problem; nothing improves.
- Candidate count set too low; the relevant chunk is cut before reranking.
- LLM reranker returns inconsistent orderings run to run.
- Reranker latency pushes p95 past the interactive budget.
- Reranker trained on web QA misjudges domain-specific relevance; not evaluated on own data.
Tradeoffs
One extra model call per query; the before/after ranking is easy to log and inspect.