Designing a Recommendation System
Events → candidate generation → features and embeddings → ranking → serving → feedback. A two-stage latency budget, a loop in which the model writes its own training data, and an offline metric that measures agreement with the previous policy.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How is a recommendation system designed end to end, and why can its offline ranking metric improve every quarter while the product gets worse?
Our home screen shows a static list of popular templates. Product wants it personalised: each account should see the items most likely to be useful to it, from a catalogue of tens of thousands, inside the page-load budget. Then they want to know it is working.
Train a click model on account and item features over the logged impressions, score every item for every account at request time, show the top few. The offline ranking metric on held-out clicks improves with every retrain.
Scoring tens of thousands of items per request with a ranker that takes a millisecond each is tens of seconds; the page times out and the fallback popularity list is what most users actually see.
- Scoring tens of thousands of items per request with a ranker that takes a millisecond each is tens of seconds; the page times out and the fallback popularity list is what most users actually see.
- Popular items were shown most, so they have the most clicks, so the model learns they are best, so it shows them more. Three months in, a tiny fraction of the catalogue accounts for nearly all impressions and new content is never surfaced.
- A challenger ranker beats the champion on held-out clicks and loses in the A/B test, because the held-out clicks are on items the champion showed; the challenger is being scored on how well it agrees with the champion.
- Click-through rises every quarter. Retention among heavy home-screen users falls. Nobody has put the two graphs on the same page.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict, for a given account at a given moment, which items from the catalogue the account will engage with — and, behind that, which items will lead to the account adopting features it keeps paying for. The click label is available immediately and is a proxy; the retention label is the goal and arrives months later (Recommendation Systems).
- The decision is which few items to show in which slots. Everything the model learns next comes from what was shown, which makes this system different in kind from a classifier (Feedback Loops).
- Product events — impressions, clicks, dwell, ignores — flow through the data platform. One training example is one impression: an account, an item, a slot, a timestamp, and whether it was clicked. Impressions that never happened, which is almost all of the catalogue for almost all accounts, are not in the data at all.
- Item metadata (category, author, age, text) and account context (industry, plan, team size, connected integrations) exist independently of interactions and are what the system has for accounts and items with no history (Cold Start).
- Every logged click was generated under some previous recommendation policy, in a position that itself affects click probability. The dataset is a record of the old system's behaviour as much as of user preference (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The architecture is two stages because the latency budget cannot afford one. Candidate generation reduces the catalogue to a few hundred items per account using cheap methods — an approximate nearest-neighbour search over embeddings (Embeddings, Cosine Similarity), collaborative filtering (Collaborative Filtering), content-based rules, popularity within segment (Candidate Generation vs Ranking). Ranking then scores those few hundred with a model that can afford richer features because it runs a hundred times less often.
- The feedback loop is structural, not a bug: what the system shows determines what is clicked, which determines the next training set. A model can only learn about items it has exposed, and an item never shown has no evidence, which the naive dataset records as zero interest. Without logging the propensity — the probability the item was shown — and without reserving some exposure for exploration, the loop converges on whatever it started showing (Exploration vs Exploitation).
- The offline ranking metric on logged clicks estimates agreement with the logging policy, not quality. It is useful for filtering candidates worth testing and dangerous as a promotion criterion; position-debiasing and inverse-propensity weighting narrow the gap without closing it (Offline vs Online Evaluation).
Two stages, one latency budget, one loop
The diagram is the reference shape. The dashed sense of the last edge — feedback flowing back into events — is the part every other ML architecture lacks: this system's serving output is its next training input. Candidate generation is drawn as a separate stage because the budget forces it; a few hundred candidates scored with a rich model costs less than tens of thousands scored with a cheap one, and gives better results.
Ownership follows the general architecture (ML System Design Architecture): events and the platform are Data Engineering (The Event Log); the vector index and the API path are shared with Backend and the database domain (Vector Search: Embeddings, Similarity and ANN); the two models, the propensity logging and the evaluation are this domain. The capstone at /ml/capstone injects the failures — popularity bias, cold start, a ranker that blows the budget, a loop that optimises clicks against retention, and an offline win that loses online.
The offline metric measures the previous policy
Held-out clicks were generated by the champion: they are clicks on items the champion showed, in positions the champion chose. A challenger that ranks the champion's favourites highly scores well on them regardless of whether its own recommendations would be clicked, and it is never scored on items the champion never surfaced. That is not a noisy metric; it is a metric for a different question.
The cost of closing the gap is an online experiment with a real holdout, which takes weeks and exposes users to a challenger that might be worse. The cost of not closing it is shipping on agreement with the past.
The challenger improves the ranking metric on held-out logged clicks by a clear margin over the champion, consistently across retrains.
In the A/B test the challenger loses on feature adoption and on the retention proxy, and click-through is flat; the team proposes that the test must be broken.
- 1The held-out clicks are on the champion's impressions, so the metric rewards agreeing with the champion, which the challenger does more precisely than the champion itself.
- 2Position bias credits items for where the champion placed them; the challenger inherits that credit without earning it.
- 3The challenger ranks the concentrated popular set higher still, which the offline data rewards and the retention proxy punishes.
Cold start and the exposure assumption
A collaborative model represents an account as a vector learned from its interactions, and an account with none has no vector. The fallback is a decision — the population prior, a content-based prior from account context, an exploration phase — and it is made by default if it is not made explicitly. The same applies to every new item in the catalogue.
Underneath both cold start and popularity bias is one assumption: that every item the system should learn about receives enough exposure to generate evidence. That assumption is violated by construction unless exposure is deliberately allocated, and the violation is silent because the aggregate metric is dominated by warm entities.
Items and accounts the system should learn about accumulate enough logged impressions that their click evidence is informative, and impressions are logged with propensity so that rare exposure can be weighted.
holds when A reserved fraction of slots goes to under-exposed items; new items enter the candidate set through content features rather than waiting for interactions; propensity is logged and used in training.
breaks when Exploration slots are removed as an "optimisation"; the candidate generator only surfaces items with interaction history; the propensity field is dropped from the log to save storage.
respond Restore exposure before retraining. A retrain on data without exploration teaches the model that unexposed items are uninteresting, and the next retrain teaches it more firmly.
1def training_weight(row):2 # row: one logged impression, joined to the click that followed (or not)3 # propensity: probability the serving policy showed this item in this slot4 # A rarely shown item that was clicked is evidence worth more than5 # a click on an item shown to everyone.6 if row["propensity"] is None:7 raise ValueError("impression logged without propensity - cannot debias")8 w = 1.0 / max(row["propensity"], 0.02) # clip to bound the variance9 return min(w, 50.0) # and cap it, at the cost of some biasThe two constants are the whole trade-off: the clip and the cap bound the variance of the estimate at the cost of bias toward the logging policy. Set them too tight and you are back to training on raw clicks; too loose and one rare impression dominates a batch.
How to build it
Most important first.
- Split the latency budget explicitly between stages: candidate generation, which can be precomputed per account in batch or run against a vector index (Vector Search: Embeddings, Similarity and ANN), and ranking, which runs online over the candidate set as one batched call (Inference Batching).
- Log the impression, the position and the propensity with every event, train on impression-weighted data, and reserve a small fraction of slots for exploration so that new items and uncertain items accumulate evidence.
- Give cold accounts and cold items a content-based path — account context and item metadata feed the candidate generator directly — and measure the cold slice separately, because the aggregate metric is dominated by entities the model already knows.
- Make the promotion gate require an online result on a business metric, with a persistent holdout of accounts on a non-personalised screen, so the system's effect on retention can be measured against something (A/B Testing Models).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Online, on the exposed population against the holdout: the business metric the recommendations are supposed to move — feature adoption that predicts retention — and click-through as a diagnostic beside it, never alone.
- Coverage and new-item exposure as first-class metrics: what fraction of the catalogue received impressions this week, and how quickly a new item accumulates enough exposure to be ranked on evidence.
- Offline ranking metrics on held-out logged clicks are a filter for which challengers to test. They are not a measurement of the system, and a rising one with a flat business metric is a warning rather than a success.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every impression is logged with its position and propensity, so training can weight by exposure and evaluation can debias by position.
- A stable holdout of accounts sees a non-personalised screen, so the system's effect on the business metric can be estimated.
- The candidate generator's recall is high enough that the items the ranker would have ranked highest are usually in the candidate set — a ranker cannot rank what it never sees.
- The latency budget per stage is met at the tail, not the median, because the fallback on timeout is the popularity list and its share of traffic is a quality metric.
- Offline: evaluate challengers on the exploration slice, where items were shown at random, alongside the logged-click metric; a challenger that wins only on the logged clicks is suspect.
- Online: A/B against the champion on the business metric with the holdout as the baseline; measure the fallback rate and the per-stage tail latency during the test.
- Over time: track coverage, new-item time-to-exposure, and the divergence between click-through and the retention proxy; a widening gap is the loop optimising the proxy.
What can go wrong
- The exploration slots are implemented and then quietly removed by a well-meaning optimisation that noticed they had lower click-through.
- Candidate generation is precomputed nightly and the ranker is online, so a new item is invisible until tomorrow whatever the ranker thinks; the freshness of the two stages differs and nobody documented it.
- The ranker is promoted on ranking quality alone; it scores candidates one at a time through a batching layer tuned for throughput, and the page takes over a second while the GPU sits idle.
- The holdout is eroded by product features that personalise elsewhere; after a year the "non-personalised" control has been personalised by three other systems.
- Exploration costs clicks today for evidence tomorrow, and the cost lands on a metric someone is measured on this quarter.
- A two-stage system has two freshness values, two models to version and a join between them that has to be logged; a single-stage system would be simpler and cannot meet the budget.
- The holdout is a fraction of accounts deliberately given a worse experience so that the effect on the rest can be measured — a cost that is easy to argue away and impossible to recover once it is gone.
- "Offline ranking metric up, ship it." The metric is computed on clicks the previous policy generated; improvement on it is agreement with the old system. The A/B test is the evaluation; the offline number decides what to test.
- "More data will fix the popularity bias." More data from the same loop is more of the loop, and a larger model learns the concentration more precisely. What fixes it is changing what is logged and what is exposed.
- "The GPU is idle so we need fewer GPUs." Idle because it is waiting on a batching timer; the request is slow for the same reason. The fix is the batching configuration, not the fleet size.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- TASK-SPECIFICThe feedback loop and the two-stage split are specific to systems that choose what to show from a large catalogue; a classifier that scores each input independently has neither, and its offline metric is correspondingly more trustworthy.
- SCALE-SPECIFICWith a catalogue of a few hundred items a single ranker over everything meets any latency budget and candidate generation is unnecessary; the two-stage design is forced somewhere in the thousands and is standard in the millions.
- SIMPLIFIEDAny counts or percentages in this lesson are illustrative and describe the shape of the argument; they are not measurements of a real catalogue or product.
Where the depth lives
This domain teaches the model and hands the rest off by name.