Designing Search Ranking
Query → candidate retrieval → ranking model → results. Lexical and embedding retrieval, learning-to-rank from click labels that carry position bias, NDCG-style evaluation at concept level, a latency budget per stage, and interleaving for the online test.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How is a search ranking system designed so that click data trains a ranker without teaching it that whatever is at the top is what people want?
Our in-product search returns results by keyword match and users say it "never finds the right thing". Product wants results ordered by how likely each is to be what the user meant, learned from what people click — inside a budget that keeps the results page instant.
Retrieve by keyword match, train a click model over query-document features on the click logs, sort the retrieved set by the predicted click probability. Offline NDCG on held-out clicks improves over the keyword baseline.
The click model learns that position one gets clicked. Its top feature is whatever correlates with having been ranked first by the old system, and its predictions reproduce the old ranking with a new justification.
- The click model learns that position one gets clicked. Its top feature is whatever correlates with having been ranked first by the old system, and its predictions reproduce the old ranking with a new justification.
- Keyword retrieval never surfaces documents that use different words for the same thing, so the ranker never sees them and the offline metric cannot penalise their absence. The users who said search "never finds the right thing" meant exactly those documents.
- A cross-encoder ranker scores each candidate with a forward pass; over a few hundred candidates it blows the page budget, and the fallback — keyword order — is what most users see.
- An A/B test on click-through shows the new ranker winning; the click-through rose because the ranker learned to put clickable titles first, and task completion — the thing search is for — did not move.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- For a query and a candidate document, predict a relevance score such that ordering candidates by it puts what the user wanted first. The label is implicit — a click, a dwell, a subsequent action — and is a noisy, position-biased proxy for relevance (Ranking).
- The decision is an ordering of a few hundred candidates, and the metric that maps to it is one that rewards relevant items near the top and discounts the rest — an NDCG-style measure — not a per-document classification accuracy.
- One training example is a query, a document that was shown, its position, and the user's response. The set of documents shown was chosen by the previous retrieval and ranking system, so the data contains only what the old system surfaced (Selection Bias).
- Click probability depends on position independently of relevance: users click the first result more because it is first. A dataset of raw clicks encodes the old ordering as if it were preference, and a ranker trained on it learns to reproduce the old ordering (Feedback Loops).
- Documents have text, metadata and — in this product — an embedding produced by an encoder (Embeddings); the query has the same. Lexical matching and embedding similarity retrieve different candidates and fail differently.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Retrieval and ranking are separated for the same reason as in recommendation (Candidate Generation vs Ranking): the budget cannot afford a rich model over the whole corpus. Retrieval runs two cheap methods in parallel — a lexical index for exact terms and identifiers, and an approximate nearest-neighbour search over embeddings for meaning — and unions them. Each covers the other's blind spot; neither is sufficient alone (Dense, Sparse & Hybrid Retrieval).
- Learning-to-rank trains on relative judgements within a query rather than absolute labels: pairs or lists of documents for the same query, with a loss that rewards putting the clicked one above the skipped one. Position bias is handled by modelling it — a click is the product of examination probability (from position) and relevance (from the document) — and training the relevance part with the examination part factored out, or by weighting clicks by inverse examination propensity.
- NDCG-style evaluation discounts each result's relevance by a function of its rank, so a relevant document at position one counts fully and at position ten barely; it is the metric shape that matches an ordering decision. Computed on held-out clicks it inherits their bias; computed on human-judged relevance for a sample of queries it is closer to ground truth and much more expensive.
- Online evaluation for ranking has a tool the rest of the domain lacks: interleaving. Two rankers' results are merged into one list per query and credit is assigned by which ranker contributed the clicked items, which needs far less traffic than a split-traffic A/B test to reach a decision, because every user sees both (A/B Testing Models).
Two retrievers, one ranker, one budget
The shape is the recommendation architecture with a query in place of an account, and the same reason for the split: the ranker cannot afford the corpus. The two retrievers run in parallel and the union is what the ranker sees. Document-side work — embeddings, static features, the index — is done ahead of time; the query side is the only thing computed inside the request.
Retrieval infrastructure lives with the database and agentic domains — the ANN index (Vector Search: Embeddings, Similarity and ANN), lexical indexing (JSONB, Full-Text Search and Extensions), chunking and hybrid retrieval as a RAG concern (RAG Overview, Ingestion: Parsing & Chunking) — and the encoder is this domain's (Embedding Training). The ranker and its evaluation are this domain's; the retrieval quality evaluation for a RAG pipeline is agentic (The Boundary With Agentic Engineering).
Clicks are a label for "was on top"
A click at position one is evidence that the document was examined and judged relevant; a non-click at position eight is mostly evidence that the user never looked. Raw clicks fold examination and relevance into one bit, and a ranker trained on that bit learns examination — which is to say, it learns the previous ranker.
The device below states this as a leakage case, because it is one: the previous ranker's decision reaches the model through the position, and the offline metric on the same clicks cannot see it. The fine case is real too — position is legitimately informative about examination, and the fix is to model it rather than to forbid it.
looks like A strong, sensible feature: documents that were ranked highly before tend to be clicked, so historical rank predicts clicks with excellent validation numbers.
why it leaks The label — a click — was generated partly by the position the previous system chose, so the feature carries the old ranker's decision into the new one. The model is being trained to agree with its predecessor.
fix Train with an examination model that factors position out of the click, or weight clicks by inverse examination propensity estimated from the randomised slice; evaluate on the randomised slice and the judged set, not on raw held-out clicks.
Latency per stage and the online test
The results page has one budget and five stages spend it. The budget is allocated at the tail, and the allocation is a design decision with a quality consequence: a ranker given more of the budget can be richer; retrieval given more can return more candidates. The fallback when a stage exceeds its share is keyword order, and the fraction of queries served that way is a quality metric, because a ranker that timed out ranked nothing.
Online evaluation is where ranking has an advantage. Interleaving merges two rankers' lists for the same query and credits whichever contributed the clicked item, so every user sees both and the comparison is paired; the traffic needed to reach a decision is a fraction of a split A/B. The A/B test still follows, on task completion, because interleaving measures preference between orderings and not whether search is doing its job.
- 1Query encoding
Encode the query once for embedding retrieval; the document side is precomputed
fails by Encoder version differs from the one that indexed the documents — retrieval returns confident nonsense
- 2Lexical + ANN retrieval (parallel)
Each returns a bounded candidate list within its share of the budget
fails by ANN recall degrades after an index rebuild; nothing downstream can recover the missing documents
- 3Union + feature assembly
Merge, dedupe, attach precomputed document features and query-document features
fails by A feature lookup to a cold cache adds a tail nobody budgeted
- 4Ranking
Score the whole candidate set as one batch
fails by Candidates scored one at a time through a batching layer tuned for throughput; the GPU idles and the page waits
- 5Response + logging
Return the ordering; log position, slice membership and the eventual completion
fails by Position not logged — the next ranker cannot be debiased
The failure column is the same lesson five times: each stage can fail without an error, and the ranker's offline metric cannot see any of them. Per-stage tail latency and per-stage recall are the monitors, and the fallback rate is the headline.
How to build it
Most important first.
- Two retrievers, one union, one ranker: lexical for terms and identifiers, embedding search for paraphrase, merged into a candidate set of a few hundred, with recall of the union measured against judged relevance so that the ranker is not blamed for what retrieval never surfaced.
- Log position and examination context with every impression, and train the ranker with a position-aware objective; hold out a small randomised slice — top results shuffled within the first few positions — to estimate examination bias and to give an unbiased evaluation set.
- Budget per stage at the tail: lexical retrieval, embedding retrieval, merge, feature assembly, ranking, response. Score the candidate set as one batch. Precompute document-side features and embeddings; only the query side is computed at request time (Latency Breakdown).
- Promote on interleaving against the champion, then confirm on an A/B test with a task-completion metric, with offline NDCG as the filter for what to test rather than the criterion for shipping.
- Keep a judged relevance set — a few hundred queries with human labels, refreshed as the corpus and the queries change — as the offline anchor that click data cannot provide.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Online: task completion after search — the user opened, used or acted on a result — against the champion in interleaving and then in an A/B test. Click-through is a diagnostic beside it; a rise in clicks with flat completion is the ranker learning clickability.
- Offline: NDCG-style on the judged set and on the randomised slice; recall of the retrieval union at the candidate-set size against the judged set. NDCG on raw held-out clicks is reported with its bias named.
- Per-stage tail latency and the fallback rate — the fraction of queries served in keyword order because a stage timed out — as quality metrics, because a ranker nobody sees has no quality.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Query and document embeddings are produced by the same encoder version, and a change to either side re-indexes the other before serving.
- A randomised slice of impressions continues to exist so that examination bias can be estimated and an unbiased evaluation set refreshed.
- Retrieval recall at the candidate-set size is high enough that the documents the ranker would rank first are in the candidate set; measured against judged relevance, not against logged clicks.
- The judged relevance set reflects the current query distribution and corpus, and is refreshed when either shifts.
- Offline: NDCG on the judged set and the randomised slice for every challenger; retrieval recall of the union at the candidate-set size; a check that the challenger's top feature is not a proxy for the old rank.
- Online: interleave the challenger against the champion until the credit difference is significant; then an A/B test on task completion with per-stage latency and fallback rate monitored throughout.
- Over time: encoder version parity between the query and document sides; ANN recall against the judged set after every index rebuild; the gap between click-through and task completion per quarter.
What can go wrong
- The embedding retriever is rebuilt with a new encoder version, the document embeddings are re-indexed, and the query encoder is not updated in the same deploy; queries and documents are now in different spaces and retrieval returns confident nonsense (Embedding Drift).
- The randomised slice is removed because a stakeholder saw shuffled results; position bias can no longer be estimated and the ranker drifts toward the old ordering with each retrain.
- The corpus grows and the ANN index's recall degrades quietly; the ranker still ranks what it is given, offline NDCG on logged clicks is unchanged, and users keep not finding the new documents.
- A ranker trained on clicks from a power-user segment is deployed to everyone; NDCG on the judged set, which was also built by power users, agrees with the model and disagrees with the population (Evaluation Slices).
- A randomised slice shows a few users a worse ordering so that the system can learn what is relevant rather than what was on top; the cost is real and lands on a metric someone watches.
- Human-judged relevance is expensive, slow, and represents the judges' idea of relevance; click data is free, fast, and represents the old ranking. Neither alone is enough.
- Two retrievers double the infrastructure and the failure modes — an ANN index with its own recall, a lexical index with its own sync (Keeping a Search Index in Sync) — for a recall gain that keyword-only search cannot reach.
- "Offline NDCG improved on held-out clicks, ship it." The clicks were generated under the champion's ordering; NDCG on them rewards agreeing with the champion. Interleave it.
- "Embedding search replaces the keyword index." It replaces it for paraphrase and loses it for exact identifiers, product codes and rare terms — the queries where users most know what they want.
- "Click-through is up, so search is better." Search is for finding; the metric is whether the user found. A ranker can raise clicks by preferring clickable titles and lower task completion at the same time.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- TASK-SPECIFICLearning-to-rank objectives, NDCG-style evaluation and interleaving are specific to ordering decisions; a classifier scoring each document independently would use a different loss, a different metric and a split-traffic test.
- DATA-SPECIFICPosition bias is severe in a vertical results list and weaker in a grid or carousel where examination is less ordered; the examination model in the ranking objective has to match the surface it was logged from.
- CONTESTEDA serious position holds that human-judged relevance sets are the wrong anchor: judges do not know what the user meant, the sets lag the query distribution, and a well-debiased click model on a large randomised slice is both cheaper and closer to users. The judged-set camp answers that debiasing assumes an examination model that is itself estimated from clicks, and a small judged set is the only signal with no loop in it.
Where the depth lives
This domain teaches the model and hands the rest off by name.