Recommendation Systems
A recommender is a loop, not a model: events feed candidate generation, candidates are ranked, the ranking decides what users see, and what users see decides the next batch of events.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What is a recommendation system made of, and why is its offline evaluation less trustworthy than a classifier's?
A streaming product has a home screen with twenty rows and a catalogue of forty thousand titles. The product lead says: "people scroll past the first row and leave. Show each person the things they will actually watch, and tell me how we will know it worked."
Treat it as classification. Build a table of (user, item, clicked?), train a model to predict the click, and at request time score every item in the catalogue for the user and show the top twenty. Evaluate with AUC on a held-out slice of the log.
Scoring forty thousand items per request inside a page-load budget is not possible with a model rich enough to be useful, so the system needs a retrieval stage that never appears in the notebook (Candidate Generation vs Ranking).
- Scoring forty thousand items per request inside a page-load budget is not possible with a model rich enough to be useful, so the system needs a retrieval stage that never appears in the notebook (Candidate Generation vs Ranking).
- The held-out log only contains items the old recommender showed. A model that reproduces the old recommender's choices scores well on it; a model that would have shown something better scores badly, because the better item has no click in the log (Feedback Loops).
- Offline AUC improves, the model is shipped, clicks rise for two weeks, and retention falls: the new model learned that clickbait thumbnails get clicks, and users who click on clickbait leave. The offline metric and the business metric diverged and nothing in the offline evaluation could show it.
- New titles never get shown, because a model trained on interactions has nothing to say about an item with no interactions (Cold Start).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- For a user at a moment, an ordered list of items that maximises some engagement outcome — a play, a watch of more than five minutes, a completion. The label is the outcome on an item *that was shown*; an item that was never shown has no label at all.
- The business target is retention, not clicks. Clicks are the proxy that is available the same day; retention is the number that matters and arrives over months.
- An event log: user, item, timestamp, position on the screen, action (impression, click, play, abandon). One example is one impression with what happened to it — which means the dataset only contains items a previous model chose to show.
- Item metadata (genre, language, year, cast) and user history. Both are sparse for the long tail: most items have a handful of plays, most users a handful of sessions.
- The log is produced by the current recommender. The distribution of items in it is the distribution the current model created, not the distribution of what users would enjoy.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The architecture is a funnel. User and item events are logged; candidate generation retrieves a few hundred plausible items from the catalogue using cheap signals; a ranking model scores those candidates with rich features; the top of the ranking is shown; what happens is logged as the next event.
- Each stage has a different job and a different metric. Retrieval must not miss the right item (recall over the candidate set); ranking must put the right item at the top (an ordering metric over the shown list); the whole system is judged on what users do, which only an online experiment can measure.
- The loop closes through the log: the training set for the next model is the impressions the current model produced. This is the structural difference from a classifier, where the label exists independently of the model's decision.
The architecture is a funnel that closes into a loop
Nothing about the diagram is exotic until the last edge. Events are logged, candidates are generated from them, a ranker orders the candidates, the top of the order is shown. The last edge — what users do with what they were shown goes back into the event log — is what makes this domain different from classification.
A fraud model's label exists whether or not the model runs: the chargeback happens or it does not. A recommender's label only exists for items the model chose to show. The model decides which examples the next model will have.
The offline number measures agreement with the past
A held-out slice of the impression log contains the items the previous model showed and what users did with them. A candidate model is scored on how well it orders those items. It is never scored on the items it would have shown instead, because nobody was shown them and there is no label.
So a model that reproduces the old model's preferences with slightly better ordering scores well, and a model that would surface different, better items scores badly. The offline metric is biased toward the incumbent, and the size of the bias is unknown without randomised exposure data.
The ranker's ordering metric on the held-out impression log improved clearly over the incumbent, on the same features and the same split.
Click-through rose for the first two weeks, then fourteen-day retention in the treatment arm fell below control and stayed there.
- 1The new ranker learned that high-contrast thumbnails and cliffhanger titles earn clicks, and optimised for them; users who clicked did not finish and returned less often.
- 2The offline log rewarded ordering the incumbent's items well and could not penalise the new model for what it stopped showing — the slower titles that retained people.
- 3Position bias in the log made whatever the incumbent placed first look good, and the new model inherited that preference as if it were a property of the items.
What must stay true after the model ships
The model was trained on a log in which a click meant a certain thing at a certain position on a certain layout. Deploying it assumes the meaning of the signal survives — and the model itself is one of the things that changes the meaning.
The assumption to make explicit is that the training data is not entirely a product of the model being trained. Without an unshaped slice of traffic, the loop has no anchor and the drift is invisible to every offline check.
A fraction of impressions are chosen independently of the deployed model, so the next training set contains evidence about items the model would not have shown.
holds when A randomised or held-out exposure slice runs continuously, is logged with the same schema as model traffic, and is used — with propensity weights — in the next evaluation.
breaks when The slice is switched off to recover engagement; the slice is logged but excluded from training as "noise"; the slice exists but is so small that long-tail items never appear in it.
respond Restore the slice before retraining. Retraining on a log with no independent exposure reinforces the loop with newer weights.
Record (user, item, clicked) for every click. Treat absence as a negative when training.
Record every impression with (user, item, position, model_score, propensity, action). Negatives are unclicked impressions; unseen items are unknown, not negative.
Clicks alone cannot distinguish "shown and rejected" from "never shown". Position and propensity are what allow the next evaluation to correct for what the model chose to show, which is the only way an offline metric can be made less biased toward the incumbent.
How to build it
Most important first.
- Start from the decision, not the model: which surface, how many slots, what action counts as success, and whether that action predicts the retention the business wants (Decision Before Model, Business Metrics vs Model Metrics).
- Separate retrieval from ranking from the first version, even when both are simple, so the latency budget and the evaluation can be reasoned about per stage.
- Log impressions with position and the propensity the model assigned, not just clicks. Without impressions there is no negative signal; without propensities there is no way to correct for what the model chose to show.
- Reserve a small share of traffic for randomised or held-out exposure from day one. It is the only data that is not shaped by the model, and it is what makes the next offline evaluation honest (Exploration vs Exploitation).
- Ship through an online experiment measuring the business outcome, with the offline metric as a gate for what is worth testing, not as the decision (A/B Testing Models).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The online business metric on a randomised holdout — retention, watch time, or whatever the product actually optimises. This is the only number that maps to the shipping decision.
- Per-stage offline metrics as gates: recall@k of the retrieval stage against items the user later engaged with; an ordering metric for the ranker on logged impressions, ideally weighted by inverse propensity.
- Do not treat click-through rate as the objective. It is the proxy most easily inflated by showing things people click on and regret.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Users still act on what is shown in roughly the way the training log recorded — a click still means what it meant, and the surface layout has not changed the meaning of an impression.
- The catalogue the ranker sees is the catalogue retrieval can reach; an item that retrieval never returns is invisible to the whole system regardless of how the ranker would score it.
- Some fraction of traffic is not shaped by the model, so the next training set contains evidence the model did not create.
- The proxy label the model optimises still correlates with the business outcome it was chosen to stand in for.
- Offline: per-stage metrics on a temporally held-out log, with an explicit check that the retrieval stage's recall covers the items users engaged with on other surfaces or through search.
- Online: an A/B test against the current system on the business metric, run long enough for retention to move, with the randomised slice kept out of both arms.
- Over time: catalogue coverage and impression concentration per week; a rising share of impressions on a shrinking set of items is the loop tightening before any metric drops.
What can go wrong
- Catalogue coverage collapses: a few hundred popular items receive nearly all impressions, the log fills with them, and the next model is more certain they are what everyone wants (Collaborative Filtering).
- Ranking latency creeps as features are added until the page misses its budget; the fallback shows popular items and the metric quietly regresses to the baseline (Serving Fallbacks).
- A holdout that was meant to be a control group is "temporarily" given the new model to hit a quarterly number, and the only unbiased measurement in the system is gone.
- The impression log drops position, so a click at slot one and a click at slot fifteen are treated as the same evidence, and the model learns that whatever was at slot one is good.
- A two-stage system is two models, two evaluations, two sets of features and a boundary where an item can be lost — against a single model that cannot meet the latency budget.
- Randomised exposure costs engagement on the slots that carry it; the price is real and continuous, and it buys data quality that shows up only in the next model.
- Online experiments on retention take weeks and constrain how many changes can be tested; the offline gate is faster and wrong in a specific, predictable direction.
- "Offline AUC improved, so ship it." In a recommender the offline set was produced by the old model, and a higher offline score can mean the new model agrees with the old one more, not that it recommends better. The shipping decision is an online one.
- "Clicks went up, so it works." Clicks are the proxy; retention is the target. A model that learns what people click on and regret raises the first and lowers the second.
- "We have forty thousand items, so the model has plenty to learn from." The model can only learn from items that were shown. Most of the catalogue has never been shown to most users and contributes nothing.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe loop structure — the model shapes the log it is trained on — holds for any system that selects what to show and learns from the response, whether the items are films, products, jobs, ads or search results.
- SCALE-SPECIFICWith a catalogue of a few hundred items a single ranking model can score everything per request and the retrieval stage is unnecessary; the funnel is forced by catalogue size and latency budget, not by principle.
- DOMAIN-SPECIFICIn media and e-commerce the engagement proxy is cheap and the business outcome is slow; in job or housing recommendation the outcome is rarer, the cost of a bad recommendation is borne by the user, and fairness across items and users becomes a first-order requirement.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Product analytics — the experiment design that measures retention rather than clicks, and the statistics of a multi-week A/B test, are an experimentation discipline this domain uses without owning.