Ranking
The output is an order, judged by what sits at the top. The label is usually a click, which was produced by the previous ranking — so the model learns the old order as much as relevance.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The product shows a list and the model decides the order. What is the label, where did it come from, and what does the model learn about positions that were never shown?
A marketplace search lead says: "Our search results feel wrong. We have two years of queries and clicks. Train a model to put the best listings first."
Treat each shown pair as a row: clicked or not. Train a classifier on click, sort listings by predicted click probability, and evaluate with click AUC on a held-out set of queries. The best listings will have the highest predicted click.
The model learns that position predicts click — the strongest signal in the data — and reproduces the old order. Held-out click AUC is excellent because the held-out clicks came from the same order.
- The model learns that position predicts click — the strongest signal in the data — and reproduces the old order. Held-out click AUC is excellent because the held-out clicks came from the same order.
- Listings the old ranker never surfaced have no positive labels, so the new ranker never surfaces them either. The catalogue the users can find is frozen at what the previous system chose.
- Pointwise click probability is optimised per row and says nothing about the list. Two listings with similar scores may be ordered by noise, and the metric never sees the order.
- When the new ranker ships, click-through rises slightly and purchases per search do not move, because the clicks moved among items the old ranker already showed.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Order the candidate listings for a query so that the ones the user wants are at the top. The label is implicit: a click, an add-to-cart, a purchase — none of which is "relevance", each of which is a user action taken on the results they were shown.
- The decision is the order on the page, and the business outcome is purchases per search. The output is consumed as a list, so the metric must be about the top of it, not about each item in isolation (Designing Search Ranking).
- One example is one query-listing pair with features of the query, the listing, their match, and the user action. Pairs come only from listings that were shown, in the positions they were shown.
- Clicks concentrate on the top positions regardless of quality, because users look there first. The label carries the old ranking's position inside it.
- Listings that were never shown for a query have no pairs and no label; the training set is silent about the part of the catalogue the old ranker hid.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Pointwise ranking scores each pair independently and sorts. Pairwise ranking learns from preferences — for this query, item A was preferred to item B — and optimises the fraction of correctly ordered pairs. Listwise ranking optimises a metric over the whole list, such as NDCG, which weights correct ordering near the top far more than correct ordering at rank forty (Candidate Generation vs Ranking).
- NDCG-type metrics encode the product truth: the user sees the top few positions and rarely the rest, so the cost of a wrong order decays with position. A metric that treats every position equally rewards fixing the bottom of the list, which no user reads.
- Implicit labels are biased by presentation. A click is evidence of relevance conditional on the item having been shown at that position; the probability of examination falls with position, so an unclicked item at rank thirty is weak evidence of anything. Correcting for this — inverse-propensity weighting, position as a feature at training and neutralised at serving, interleaving experiments — is what separates a ranker that learns relevance from one that learns the old order (Feedback Loops).
The label was produced by the old order
A click is a user action on a page the previous ranker built. The item was shown at a position, the user examined some positions, and clicked or not. The label therefore contains three things — whether the item was relevant, whether it was examined, and whether it was shown at all — and only the first is what the new ranker should learn.
The training set is silent about anything the old ranker did not show, and biased toward whatever it put on top. A model trained naively on it learns the old ranker's preferences with high offline confidence.
Pointwise, pairwise, listwise — and where the metric lives
Pointwise treats each pair as a row and sorts by score; it is simple and its metric is about rows. Pairwise learns which of two items should come first and its metric counts correctly ordered pairs everywhere in the list. Listwise optimises a position-weighted metric over the list, which is where the user actually is.
The metric that matches the product is position-weighted. NDCG discounts each position's gain by its rank, so getting the first three right matters and getting rank forty right barely registers. That is the correct shape for a list the user reads from the top.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Pointwise click classifier | Simple to train, serve and explain; learns position bias unless it is modelled out; metric lives on rows, not on the list. | ||||||
| Pairwise preference model | Learns from within-query preferences, which cancels much of the query-level bias; treats a swap at rank forty like a swap at rank one. | ||||||
| Listwise, position-weighted objective | Optimises where the user looks; harder to train and debug; needs debiased labels or the objective optimises the old order more precisely. | ||||||
| Any of the above + exploration slice | The only option whose training data contains items the current ranker hides; costs a permanent slice of worse results. |
caveat The scores cannot express that the last row is not an alternative to the first three but a precondition for any of them improving beyond the old ranker, nor that "quality" here is an offline list metric on biased labels and the real quality is purchases per search in an online test.
Treat every shown (query, listing) pair as a row, predict click, and report AUC on a held-out set of queries.
Compute NDCG at a small k on held-out queries with inverse-propensity-weighted labels, use it to pick a candidate, and confirm with interleaving on purchases per search.
Click AUC is maximised by predicting the old order, because position is the strongest predictor of a click; a position-weighted metric on debiased labels rewards ordering by relevance where the user looks, and the online test measures the outcome the list exists to produce.
What must remain true about the label
The ranker depends on a chain: users examine the page in a known way, a click means what it meant, and the training set includes items the ranker would not choose. A page redesign, a change in what a click leads to, or a switched-off exploration slice breaks one link and the offline metric does not notice.
The monitor is on the label itself: purchase-per-click by position, and the share of surfaced items that are new to the ranker.
The implicit label still tracks the business outcome at the training-time rate per position, users still examine the page as the debiasing assumes, and the training set still contains labels for items outside the current ranker's choices.
holds when Purchase-per-click by position is stable; the page layout is unchanged or the propensity model has been re-estimated; the exploration slice is live and its labels flow into training.
breaks when A layout change moves the fold; a promotion makes clicks cheap and purchases rare; the exploration slice is switched off for a peak season.
respond Re-estimate examination propensities after any layout change; restore exploration before retraining; if the label's meaning moved, change the label, not the model.
How to build it
Most important first.
- Choose the label to match the outcome: a purchase or a dwell-weighted click over a raw click, because raw clicks reward attractive thumbnails and misleading titles (Target Definition).
- Model examination separately from relevance: include position at training time so the model can attribute clicks to it, and serve with position fixed to a constant so the relevance part is what sorts (Label Leakage).
- Reserve a small share of traffic for randomised or exploratory ordering, so the training set contains labels for items the current ranker would not show (Exploration vs Exploitation).
- Evaluate with a position-weighted list metric on held-out queries, and confirm online with an interleaving test or an A/B on purchases per search, because offline list metrics are computed on biased labels (Offline vs Online Evaluation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Purchases per search, or the business outcome the list is for, in an online comparison. That is the decision number; no offline metric substitutes.
- NDCG at a small k on held-out queries with a debiased label, for comparing candidate rankers before an online test. Its absolute value means little; its relative movement means something.
- Click AUC on shown pairs is the number that looks relevant and is not: it rewards reproducing the position bias in the labels and is highest for the model that copies the old ranker.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The implicit label still tracks the outcome the business wants — a click still leads to a purchase at roughly the rate it did at training time — which the purchase-per-click ratio per position can check.
- The examination model still describes how users read the page: a redesign that shows more results above the fold changes the position bias and the debiasing with it.
- The training data continues to include labels for items the current ranker would not show, from a maintained exploration slice.
- Offline: NDCG at small k on a held-out query set with propensity-weighted labels; a check that the model's scores do not change when the position feature is varied at serving time.
- Online: interleaving against the current ranker on a traffic slice, then an A/B on purchases per search; both must agree before a full rollout.
- Over time: purchase-per-click by position as a label-drift monitor; the share of surfaced listings that were never surfaced in the previous quarter, as a check that exploration is alive.
What can go wrong
- The exploration slice is turned off during a peak season to protect revenue and never turned back on; a year later the ranker has trained twice on its own choices.
- The purchase label is sparse and the team falls back to clicks; the new ranker optimises for listings that get clicked and returned.
- Position is included as a training feature and left as the real position at serving time, so the ranker scores the top slot higher because it is the top slot.
- Exploration traffic shows some users a worse list on purpose; it is a permanent tax paid to keep the training data honest.
- Debiased labels are noisier and lower-volume than raw clicks, so the offline metric is less stable and the model needs more data to move it.
- A listwise objective is harder to train and debug than a pointwise classifier, and its offline metric is less intuitive to product stakeholders.
- "Click AUC on held-out queries is high, so the ranker is good." The held-out clicks were produced by the old order. High click AUC means the model predicts the old order well.
- "Users click on the top result, so the top result is the best one." Users click the top result because it is on top. The label contains the position.
- "Ranking is classification per item, then sort." Sorting independent scores ignores that the metric lives on the list and that the label was conditioned on presentation.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat implicit labels are conditioned on presentation, and that a ranking metric must weight the top of the list, holds for search, recommendations, ads and any surface where a model orders what a user sees.
- DOMAIN-SPECIFICIn web search, human relevance judgements exist and pointwise models on judged pairs are a strong baseline; in a marketplace with a changing catalogue no such judgements exist at scale, and implicit feedback with debiasing is the only label there is (Recommendation Systems).
- CONTESTEDA serious position holds that position debiasing is over-engineered for most teams and that an online A/B on the business metric, run often, is the whole answer: the offline metric is a rough filter and the biases wash out in the experiment. The counter is that an A/B cannot show what the ranker never surfaces, so without exploration the experiment compares two models that both learned the old order.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Product analytics — the interleaving and A/B designs that confirm a ranker are experimentation practice; this lesson assumes a team can run them and read purchases per search with enough power.