Performance Decay
Quality over time went down. Five different causes produce that chart, only one of them is fixed by retraining, and two are made worse by it. Diagnose in order — bug, product change, feedback loop, data drift, concept drift — before touching the model.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Model quality has been falling for a month. Which of the five things that produce that chart is it, and what happens if you retrain before you know?
A churn model's precision at the operating threshold has slid for four consecutive observable weeks. The retraining job is ready to go. The engineering manager has asked one question before approving it: "What are we retraining because of?" Nobody has an answer beyond "the number went down".
Quality is falling, so the model is stale. Retrain on the most recent data and quality will recover — that is what retraining is for, and the pipeline is built.
If a feature pipeline broke, the retrain teaches the model that the broken value is normal, quality "recovers" on broken data, and the bug becomes permanent (Feature Drift).
- If a feature pipeline broke, the retrain teaches the model that the broken value is normal, quality "recovers" on broken data, and the bug becomes permanent (Feature Drift).
- If the retention team started acting on the scores, flagged users are being saved and their outcome is now "did not churn"; a retrain learns that the users the model flags do not churn, and stops flagging them (Feedback Loops). The Drift Explorer's feedback-loop scenario has every input stable and quality among flagged users falling anyway.
- If the product changed — a new cancellation flow, a pricing change — the label's meaning shifted and no retrain on the old label definition fixes it.
- If it is data drift into a region the model got wrong, or concept drift, retraining helps — with labels from the new region or after the change, which the last four weeks mostly do not contain.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict thirty-day churn; the decision is the retention call list, and the retention team acts on it.
- The number that decayed is precision on joined outcomes, per prediction week, read at label arrival (Ground-Truth Delay).
- The prediction log with features, versions and scores; the outcome join; per-feature monitors; deploy and job history; product release notes; and the retention team's intervention log, which nobody has looked at yet.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Five causes produce a falling quality line, and each has a fingerprint in the other monitors. Feature bug: a null or default spike and a same-day prediction shift. Product change: a deploy or release aligned with the drop, often with a label-rate change. Feedback loop: inputs stable, prediction distribution stable, quality among flagged entities falling, an intervention log that started when the drop did. Data drift: a feature distance alert preceding the drop by the label delay, quality falling on the drifted slice. Concept drift: nothing else moved.
- The order of diagnosis is the order of cost-of-being-wrong. A retrain on a bug or a loop makes the system worse; a retrain on a product change does nothing; a retrain on drift helps. So the cheap-to-check, expensive-to-miss causes come first (ML Incident Debugging).
- The retrain is a valid response to two of the five, and even then only with the right data window.
Five charts that look the same
Quality per prediction week, falling. The table is what to look at next, and in what order. Order matters because the first two causes are made permanent by a retrain, the third is made much worse by one, and the last two are the only ones a retrain addresses.
The response column is the lesson: each cause has a response that is not "retrain", except the two that are, and those come with a data-window condition.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Feature pipeline bug | Null or default share spike on a feature; prediction distribution moved the same day; drop aligned with a job or deploy | The model is scoring an imputed default as a real value | Fix the pipeline; do not retrain — the retrain learns the default as normal (Feature Drift) |
| Product or label change | Drop aligned with a release; base rate of the outcome moved; inputs and predictions stable | What the label means changed — a new cancellation flow, a pricing change | Revisit the target definition; a retrain on the old label learns the old meaning (Target Definition) |
| Feedback loop | Inputs stable; prediction distribution stable; quality among flagged entities falling; an intervention started when the drop did | Acting on the prediction changed the outcome for exactly the flagged population | A holdout of unflagged entities or a label that records the intervention; a retrain without it learns the intervention as the outcome (Feedback Loops) |
| Harmful data drift | Feature distance alert preceding the drop by the label delay; quality falling on the drifted slice specifically | Traffic moved into a region where the model's form is wrong | Retrain with labelled data from the new region; validate on the slice (Data Drift) |
| Concept drift | Nothing else moved; quality falling across segments; no intervention | The relationship between features and outcome changed | Retrain on post-change labels once enough exist; add proxies to detect the next one sooner (Concept Drift) |
The loop the retrain would close
In the explorer's feedback-loop scenario the retention team starts acting on the scores at week 4. Flagged users get a call; often it works. The world outside did not change; PSI is at the noise floor; the prediction mean is steady. The share of flagged users whose outcome was positive falls week after week, because the intervention is preventing the outcome the model predicted. Measured precision decays, and the decay is the intervention succeeding.
A naive retrain on this data learns that flagged users do not churn and stops flagging exactly the people the calls were saving. The validation metric on the recent window approves the change, because the recent window contains the same loop.
The queued retrain, run on the last eight weeks, reports validation precision above the current model's on the same window.
Had it shipped: the new model flags fewer of the users the retention team was successfully saving, churn among previously-flagged users rises, and the "recovered" precision is measured on a list that no longer contains the people who needed the call.
- 1The intervention log shows the retention team began working the list at the week the decay starts; flagged-user outcomes fell because the calls worked, not because the model was wrong.
- 2The retrain's validation window shares the loop, so it rewards a model that agrees the flagged users do not churn.
- 3No input monitor moved, which is the loop's fingerprint; the same fingerprint as concept drift, separated by the intervention log.
What a retrain assumes
A retrain assumes the recent data is a better description of the world than the old data. That is true for drift and false for a bug, a loop or a label change, where the recent data is a better description of the fault. The assumption is checkable — by the fingerprint — and it is the thing to check before the job runs.
The manager's question was the right one. "What are we retraining because of" has five answers, and the retrain is the response to two of them.
The data a retrain would learn from reflects the relationship the model should learn, rather than a pipeline bug, an intervention triggered by the model, or a changed label definition.
holds when Feature null and default rates are at baseline; the deploy and release history is clean around the drop; the intervention log shows no action on the model's output, or a holdout exists; the label definition is unchanged.
breaks when A feature broke and the recent window contains it; an intervention began on flagged entities; a release changed what the outcome means.
respond Name the cause on the retrain approval; for a bug or a label change, fix upstream and skip the retrain; for a loop, build the holdout first; for drift, retrain with the right window and validate on the affected slice.
What is the decay because of, and what does that make the right data window?
when Bug signature in a feature; same-day prediction shift.
cost No retrain. Fix and let the current model recover; a retrain would need the broken weeks excluded, and the fix makes it unnecessary.
when Release aligned with the drop; outcome base rate moved.
cost No retrain on the old label. Reformulate, then rebuild the training set under the new definition.
when Intervention on flagged entities began at the drop; inputs stable.
cost No retrain until a holdout or an intervention-aware label exists; then train on the holdout or condition on the intervention.
when Feature alert preceded the drop by the label delay; the drifted slice carries it.
cost Retrain with enough labelled rows from the new region; the last few weeks alone are usually too few and mostly immature.
when Nothing else moved and there is no intervention.
cost Retrain on post-change labels — which means waiting for them, or weighting the recent mature weeks heavily and accepting the variance.
How to build it
Most important first.
- Read the other monitors before the model: feature null and default rates, prediction distribution, deploy and job history, the release calendar, the intervention log (Model Monitoring).
- Slice the decay: by segment, by feature region, by flagged versus unflagged, by model version. The slice that carries the drop names the cause more often than any aggregate does (Evaluation Slices).
- Match the timeline: the drop in quality is dated by prediction week, not by observation week; align it with what changed in that prediction week, not with what changed when the number was noticed.
- Decide the response from the cause: fix the pipeline; redefine the label; build a holdout to break the loop; retrain with the right window. Only then run the job (Retraining as a Decision, Retraining Strategies).
- Write it down. A decay investigation whose conclusion is recorded becomes the next one's baseline (Model Postmortems).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Quality per prediction week on mature labels, sliced by segment and by flagged status — the decay itself, in enough detail to locate it.
- Alongside: feature null rates and distances, prediction distribution, deploy markers and intervention volume, on the same time axis, so the fingerprint can be read.
- The retrain's expected gain, estimated from what the cause is: for a bug or a loop it is negative, and that number should be on the approval request.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The monitors needed for the fingerprint — feature, prediction, outcome join, deploy history, intervention log — exist and share a time axis, so a decay can be diagnosed rather than only noticed.
- The decision to retrain is gated on a stated cause, and the pipeline does not run on the quality trigger alone without a human naming the cause.
- The label definition is stable; a product change that alters what "churn" means is treated as a formulation change, not a decay.
- Offline: for each past decay, record the cause found and whether the response fixed it; a history where every decay was answered with a retrain and half recurred is the audit that changes the process.
- Online: the decay dashboard with the other monitors on the same axis and deploy markers, so the fingerprint is visible without an investigation.
- Over time: after the response, confirm quality recovers on the slice that carried the drop, not only on the aggregate.
What can go wrong
- The fingerprint is ambiguous — feature drift preceding a drop can be harmful data drift or a bug that also moved the distribution — and the investigation has to use the pipeline logs to separate them.
- Two causes at once: a product change altered the label rate and, in the same fortnight, a feature broke; fixing one leaves a residual decay that is then misattributed.
- The investigation takes a week, the retrain would have taken an hour, and the pressure to run it is the strongest just when the cause is least known.
- A diagnosis-first process is slower than a retrain-first one, and the cases where retrain-first was right will be remembered more than the cases where it baked in a bug.
- The intervention log and the release calendar belong to other teams, and the investigation depends on being able to read them.
- Slicing quality by segment multiplies the label requirement; small segments will not have intervals narrow enough to convict.
- "Quality fell, so retrain." The Drift Explorer has two scenarios where that makes the model worse — the feature bug and the feedback loop — and one where it does nothing until the labels catch up. The answer to "quality fell" is "why".
- "The retrain improved validation, so it worked." Validation on the recent window shares whatever broke it. A retrain on a broken feature validates beautifully on the broken feature.
- "Nothing else moved, so the monitoring is fine and the model just aged." Nothing else moving is the fingerprint of concept drift or a feedback loop, and the two need opposite responses. Check the intervention log.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe five causes and their fingerprints apply to any supervised model with delayed outcomes; which cause dominates differs — feedback loops in recommendation and retention, concept drift in fraud, pipeline bugs everywhere.
- CONTESTEDA serious position holds that scheduled retraining on a short cadence makes most of this diagnosis unnecessary: the model is always fresh, drift and concept change are absorbed automatically, and the engineering time goes into the rollout pipeline instead of into investigations. That works when the labels are fast and the pipeline is trustworthy. The counter-argument is that scheduled retraining bakes in feature bugs and feedback loops on schedule, and hides them, because the validation window shares the fault; the cadence has to be paired with the fingerprint monitors or it is automation of the wrong response.
- SIMULATEDThe feedback-loop scenario — an intervention on flagged users from week 4 that prevents the outcome for a share of them, inputs stable, flagged-outcome rate falling — and the other decay shapes are the Drift Explorer's synthetic model on generated traffic, for the shape of the argument.
Where the depth lives
This domain teaches the model and hands the rest off by name.