What Can Go Wrong
Sixteen failure classes, each entering at a specific pipeline stage, most invisible to the offline metric. Learning to name them by stage is the difference between debugging a model and retraining it in the dark.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model was fine at approval and is wrong in production. Which of the sixteen ways this happens entered at which stage, and which signal would have shown it?
The recommendation team has had four incidents this year. Each was "fixed" by retraining. Two came back. Leadership wants a list of what actually goes wrong with these systems so the next incident starts with a diagnosis instead of a training job.
A model that gets worse has gone stale. Retrain on fresh data and it recovers. If it does not recover, the model needs more capacity.
Retraining fixes exactly three of the sixteen: concept drift, data drift that the model can relearn, and staleness. It leaves leakage, skew, broken preprocessing, version mismatch, wrong metric and feedback loops in place — and in the last case, makes them worse.
- Retraining fixes exactly three of the sixteen: concept drift, data drift that the model can relearn, and staleness. It leaves leakage, skew, broken preprocessing, version mismatch, wrong metric and feedback loops in place — and in the last case, makes them worse.
- Two of the four incidents "came back" because they were train/serve skew and a feedback loop. Retraining rebuilt the skew with new weights and fed the loop another cycle.
- The offline evaluation was unchanged in every incident because none of the sixteen failures that occur after the evaluation stage can show up in a number computed before it.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The running example ranks products for a user; the label is a click or purchase within the session. The target of this lesson is a vocabulary: sixteen named failures, each tied to the stage where it enters.
- The point of naming is that different failures with identical symptoms need different responses, and only one of them is "retrain".
- Each incident produced the same evidence: a business metric fell, the offline evaluation of the deployed artifact was unchanged, and the retrain produced a slightly different artifact with a similar offline number.
- Nobody had feature-distribution logs at the serving boundary, so the incidents could not be distinguished from each other after the fact.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Each failure is a violated assumption at one pipeline stage. Failures at the dataset stage (bad labels, sampling bias, leakage, imbalance) corrupt what the model learns; failures at the training and evaluation stages (over- and underfitting, wrong metric) corrupt what is believed about it; failures at deployment and inference (skew, broken preprocessing, version mismatch, latency) corrupt what it computes; failures after deployment (data, concept and feature drift, staleness, feedback loops) corrupt the fit between the model and a world that has moved.
- The stage decides the symptom's timing. Dataset-stage failures are present from day one and are invisible offline because the evaluation shares the flaw. Deployment-stage failures appear as a step change on rollout day. Drift appears gradually. Feedback loops appear one retraining cycle later.
- The stage also decides the fix. Nothing that entered before the evaluation is fixed by retraining on the same pipeline; nothing that entered at deployment is fixed by a training run at all.
Sixteen failures, by the stage they enter
The table is the lesson. Read the stage column first: it tells you when the failure became true, which tells you whether the offline evaluation could have seen it, which tells you whether retraining can fix it. Everything that enters at or before the evaluation stage shares its flaw with the evaluation and is invisible to it; everything after cannot be measured by a number computed before it.
The retrain column is deliberately blunt. "No" means retraining on the same pipeline reproduces the failure with new weights.
| Failure | Enters at | Offline metric | Retrain fixes it? |
|---|---|---|---|
| Bad labels | Dataset | Fine or misleadingly high — the evaluation shares the labels | No |
| Data leakage | Dataset / Features | Misleadingly high | No |
| Sampling bias | Dataset | Fine on the biased sample | No |
| Class imbalance | Dataset / Evaluation | Accuracy misleadingly high | No — needs a metric and threshold change |
| Overfitting | Model / Training | Training high, validation lower | Only with a capacity or regularisation change |
| Underfitting | Model / Training | Both low | Only with a capacity or feature change |
| Wrong metric | Evaluation | Optimised — for the wrong thing | No |
| Train/serve skew | Deployment / Inference | Fine — computed on training features | No |
| Data drift | After deployment | Fine at approval; unknown now | Sometimes — if quality actually moved |
| Concept drift | After deployment | Fine at approval; wrong now | Yes, with a window that covers the new relationship |
| Feature drift | After deployment | Fine at approval | Sometimes; often the fix is upstream |
| Latency problems | Inference | Not measured | No |
| Model staleness | After deployment | Fine at approval; decays | Yes |
| Broken preprocessing | Deployment / Inference | Fine | No |
| Version mismatch | Deployment | Fine — for the version that was evaluated | No |
| Feedback loops | Feedback | Fine, and improving each cycle | No — makes it worse |
Same symptom, four causes
The recommendation team's four incidents produced the same reading — click-through fell, the offline number did not move — and were four different failures. The failure table below reconstructs them by the signal that would have separated them, which is the signal that was not being logged.
Notice that the only incident retraining fixed was the concept drift, and even there the fix was the retraining *window* rather than the retraining itself. The other three needed a pipeline fix, an upstream fix and a logging fix respectively.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A catalogue migration renamed a category column; the serving path started mapping it to "unknown". | Click-through fell by a step on migration day; offline metric unchanged. | Train/serve skew via broken preprocessing at the inference stage. | Fix the mapping; retraining would have learned "unknown" as a real category and hidden the bug. |
| A seasonal shift in what users buy. | Click-through fell gradually over six weeks; feature distributions moved; outcome rate per predicted score moved. | Concept drift after deployment. | Retrain on a window covering the new season; consider a shorter cadence. |
| A new traffic source sent users with no history. | Click-through fell for one slice; overall feature distributions moved slightly. | Data drift — a population the training set under-represented. | Cold-start handling for the slice; retraining alone learns them as low-value. |
| Three retraining cycles on data where recommended items were the only items users saw. | Offline metric improved each cycle; catalogue coverage collapsed; long-term engagement fell. | A feedback loop at the feedback stage. | Log exposure, add an exploration slice, rebuild labels; stop retraining on the loop. |
Why the number did not move
The signature of this domain is an offline metric that stays where it was while the product breaks. It is worth being precise about why. The metric is a function of the artifact and the evaluation set; neither changed. Every failure that entered after the evaluation stage changed something else — the inputs, the world, the labels being collected — and no function of two unchanged things can report it.
So the offline-online gap is not a bug in the evaluation. It is what the evaluation *is*: a statement about one dataset at one time. Closing the gap costs something in every case, and the cost is the reason teams do not, until the fourth incident.
Held-out click-through of the deployed artifact re-evaluated after the incident: unchanged from approval.
Session click-through down noticeably for users arriving from the new traffic source; flat for everyone else.
- 1The new users have no history, so their feature vectors fall in a region of the input space the training set barely covered; the model extrapolates to a low score for everything.
- 2The evaluation set was drawn from the training period, before the traffic source existed, so it contains almost none of these users and cannot register the failure.
- 3A smaller effect: the new users' sessions are shorter, so even a correct ranking has fewer clicks to earn.
How to build it
Most important first.
- Keep the sixteen as a checklist to run at incident time, ordered by stage, and refuse to retrain until a stage has been named (ML Incident Debugging).
- Instrument each stage so its failures have a signal: label audits and slice metrics for the dataset stage, a replay-equivalence test for skew, feature distributions at the serving boundary for drift, action logs for feedback loops (Model Monitoring).
- For each model, write down which of the sixteen it is most exposed to given its data and deployment, and put the detector for those in place before launch (The ML Testing Stack).
- When the cause is genuinely drift or staleness, retrain — and record the diagnosis so the next incident with the same symptom is not assumed to be the same cause (Drift Is Not Failure).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For each failure class, the signal that distinguishes it: feature distributions at serving versus training (skew, feature drift), prediction distribution over time (prediction drift), outcome rate against prediction rate once labels arrive (concept drift), label rate against the model's own flagging rate (feedback loop), p99 latency and fallback rate (latency, broken preprocessing).
- The business metric tells you something is wrong. It cannot tell you which of the sixteen, and a diagnosis made from the business metric alone is a guess.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The serving-boundary feature log, the prediction log and the action log exist and are retained long enough to cover the label delay, so an incident can be diagnosed after the fact.
- The offline evaluation that approved the model is recorded with its split, date range and class balance, so a production number can be compared with it honestly.
- The detectors for the classes this model is most exposed to are running, and their alert thresholds were set against a period known to be healthy.
- Offline: for the deployed artifact, re-run the evaluation on a period after the training window and confirm the number holds; a gap that appears with time and not with rollout points at drift, one that exists from the start points at leakage or a split problem.
- Online: on rollout day, compare serving feature distributions against training; a step change is skew, version mismatch or broken preprocessing, not drift.
- Over time: when labels arrive, compare the outcome rate for treated and untreated predictions; a divergence that grows across retraining cycles is a feedback loop.
What can go wrong
- The checklist is run and several classes fit the readings — a feature drift and a skew produce similar serving-boundary signals — and the team picks the one whose fix is cheapest (ML Incident Debugging).
- Detectors are installed for the classes the team has already suffered, and the next incident is one of the other twelve.
- The drift monitor fires so often on harmless drift that it is muted, and the concept drift that matters arrives among the muted alerts.
- Naming the cause before retraining means an incident stays open longer than it would if the team simply retrained and hoped; the discipline is paid for in time-to-mitigate on the occasions when retraining would have worked.
- Sixteen detectors are sixteen alert streams, each with its own false-positive rate; the operational cost of the monitoring can exceed the cost of the model.
- The logs the diagnosis needs — feature vectors, predictions, actions, outcomes — are a storage bill and a privacy review.
- "Drift means retrain." Drift is a change in the inputs; whether the model's quality moved is a separate question, and retraining a model on drifted inputs whose quality did not move is cost for nothing. Where quality did move, the cause may still not be drift.
- "The offline metric did not change, so the model did not break." The offline metric is computed on the training distribution. Thirteen of the sixteen failures leave it exactly where it was.
- "We fixed it last time by retraining, so retrain again." Retraining hides the three failures it does not fix — skew, feedback loops, broken preprocessing — behind new weights, and they return.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe sixteen classes and the stage each enters at are independent of model family and task; a linear model on a cron job and a transformer behind an API suffer the same sixteen, at different rates.
- DOMAIN-SPECIFICWhich classes dominate depends on the domain: fraud and recommendation live with feedback loops and concept drift, medical imaging with sampling bias and label quality, forecasting with temporal leakage; the checklist is the same, the priors are not.
Where the depth lives
This domain teaches the model and hands the rest off by name.