Ground-Truth Delay
The outcome arrives weeks or months after the prediction. Every quality number on the dashboard is about the past; the architecture has to say how far past, join outcomes back by id, and use proxies honestly in the meantime.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model's quality can only be measured once the outcomes exist, and they take a quarter to arrive. What does the monitoring show for this week, and how do you avoid being reassured by last quarter?
A payments team's dashboard shows chargeback-model precision by week, and it looks fine. The fraud lead points out that a chargeback takes up to ninety days, so the most recent week with a real number is from three months ago — and the model was retrained six weeks ago. Nobody can say how the current model is doing.
Plot precision by week from the joined outcomes. Where labels are still arriving, use what has arrived so far. The chart fills in over time, and the most recent weeks are just a bit noisy.
The recent weeks are not noisy; they are biased. A transaction with no chargeback yet is counted as a negative, so recent precision is systematically understated for a fraud model — or overstated, for a model where the late-arriving label is the negative. The chart shows a decline every week that is entirely the label window closing.
- The recent weeks are not noisy; they are biased. A transaction with no chargeback yet is counted as a negative, so recent precision is systematically understated for a fraud model — or overstated, for a model where the late-arriving label is the negative. The chart shows a decline every week that is entirely the label window closing.
- The retrained model has six weeks of predictions and almost no complete labels. Its precision on the dashboard is either missing or computed on the partial labels, and either way the number that says whether the retrain was good does not exist yet.
- The Drift Explorer shows this for every scenario: the last three weeks of accuracy are null, no matter what happened. In the concept-drift scenario the relationship changes at week 6 and the first observable drop is at week 9. The dashboard was not wrong for those three weeks; it was silent, and silence read as fine.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict chargeback within ninety days. The label is defined by a window that has not closed for any transaction in the last three months.
- The decision is the review queue, today. The quality of today's decisions will be knowable in the autumn.
- Predictions logged with transaction id, model version and score. Chargebacks land in a separate table as they arrive, with the original transaction id.
- Faster signals exist: a customer dispute is filed within ten days on average, and a manual-review verdict comes in a day for the transactions that were reviewed.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A label with a maturation window is undefined until the window closes. Counting an immature example as a negative is a censoring error: the label's value depends on how long you waited. Quality computed over a mix of mature and immature examples is a mix of a measurement and a guess.
- The join that attaches outcomes to predictions is by id, and it runs continuously: each day, more outcomes for old predictions arrive, and the quality for those weeks updates. The dashboard is not a time series of quality; it is a time series of quality-as-known-at-a-date, and the two axes have to be distinguished.
- Proxies are outcomes with shorter windows that correlate with the true label. Their value is speed; their cost is that the correlation is a measured, decaying quantity, and a proxy can be moved by things that do not move the true label (Business Metrics vs Model Metrics).
Two time axes
Every quality reading has two dates: the week the predictions were made, and the date the reading was computed. A chart with only the first axis implies that the reading was available at that week, which it was not. The explorer's convention — accuracy is null until the labels arrive, and the first quality signal is reported at the week it became observable — is the honest one.
The label-lag dashboard makes the second axis visible: for each prediction week, the share of its labels that have landed. A week at 20% arrival is not a measurement of the model; it is a measurement of how the first fifth of the labels went, and for a fraud model the first fifth is not representative.
1WITH preds AS (2 SELECT transaction_id, model_version, score, served_at::date AS pred_day3 FROM prediction_log4), mature AS (5 -- a prediction is mature when its label window has closed6 SELECT p.*, (p.pred_day + :window_days) <= current_date AS is_mature7 FROM preds p8), joined AS (9 SELECT m.*, (o.transaction_id IS NOT NULL) AS chargeback10 FROM mature m11 LEFT JOIN chargebacks o ON o.transaction_id = m.transaction_id12)13SELECT date_trunc('week', pred_day) AS prediction_week,14 model_version,15 avg(CASE WHEN is_mature THEN 1 ELSE 0 END) AS mature_share,16 -- precision only over mature rows; NULL, not 0, when there are none17 sum(CASE WHEN is_mature AND score >= 0.5 AND chargeback THEN 1 END)::float18 / nullif(sum(CASE WHEN is_mature AND score >= 0.5 THEN 1 END), 0) AS precision_mature19FROM joined20GROUP BY 1, 2 ORDER BY 1;The LEFT JOIN is where the censoring error lives: an unmatched row is "no chargeback yet", and only the maturity flag turns that into "no chargeback". Compute the metric on mature rows and show mature_share beside it.
The dashboard that showed last quarter
The payments team's precision chart was correct for every week it showed a real number. The problem was which weeks those were. The retrained model had six weeks of predictions, none mature; the chart's recent points were either blank or computed on partial labels, and the eye reads a chart from the right. The reassurance came from the old model's mature weeks.
The gap between the offline number and production here is not a model failure. It is a measurement that could not exist yet, presented as if it did.
The retrained model's validation precision was higher than the old model's; the production dashboard showed stable precision by week.
The stable weeks were the old model's; the new model had no mature labels. When they matured, its precision on the segment the retrain was meant to improve was lower — a feature the retrain leaned on had been skewed in serving.
- 1No mature labels existed for the new model, so the dashboard's recent weeks were the old model's quality or partial-label guesses; the reader took continuity for health.
- 2The proxy that could have warned earlier — manual-review verdicts on flagged transactions — was not on the dashboard, because it was "not the real metric".
- 3Prediction drift on rollout day would have flagged the skewed feature without any label at all (Prediction Drift).
Designing for a late label
The architecture has three parts: a prediction log with a join key, a continuously updated outcome join with a maturity boundary, and a proxy pipeline with a correlation monitor. The rollout process then has to accept that the decisive number comes late, judge the canary on what exists, and schedule the re-evaluation.
The assumption to keep checking is that the join is complete and the window is what the design says. Both drift.
Every prediction can be matched to its eventual outcome by id, the share that match is stable, and the label window after which an unmatched prediction is a true negative is known and stable.
holds when The prediction log and the outcome table share a key that survives refunds, retries and re-issued ids; the arrival curve of labels is monitored and its tail is inside the maturity threshold; join rate per week is flat.
breaks when An outcome lands on a different id than the prediction; a processor changes its chargeback timeline; a data-retention job deletes outcomes older than the window; the join rate falls and quality appears to rise.
respond Fix the key mapping; move the maturity threshold to cover the tail; treat a join-rate fall as a data incident, and mark the affected weeks' quality as unknown rather than as measured.
| Signal | Delay | Measures | Trust it for |
|---|---|---|---|
| Prediction distribution vs reference | Same day | Whether the model is receiving what it was validated on | Rollout-day skew and pipeline bugs; nothing about quality |
| Manual-review verdict | A day | Precision on flagged transactions only | A leading indicator of precision; silent on recall |
| Customer dispute | About ten days | A correlated, earlier outcome | Early warning, with its correlation to chargeback monitored |
| Chargeback (the label) | Up to ninety days plus tail | The outcome the model predicts | The promotion decision — provisional until it arrives |
How to build it
Most important first.
- Log every prediction with an id the outcome will carry, and build the outcome join as a continuously updated table keyed by prediction and version (Prediction Logging).
- Compute quality only on mature examples — those whose label window has closed — and show the maturity boundary on the chart, so the reader sees where the measurement ends.
- Build the label-lag dashboard: for each prediction week, what share of its labels have arrived. This is the monitor that says which weeks can be read.
- Add proxy quality for immature weeks, on its own axis, with the proxy's validated correlation to the true label shown alongside, and re-validate it as true labels arrive (Concept Drift).
- Design the rollout to tolerate delayed evaluation: a canary judged on proxies and prediction distribution first, and a promotion decision that is revisited when the mature labels arrive (Canary Rollout, Champion / Challenger).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Quality per prediction week on mature labels, labelled with the week it became computable. This is the number that says whether the model was good, and it is always about the past.
- Label arrival share per prediction week — the label-lag curve — so the reader knows which weeks are measurements and which are guesses.
- Proxy quality for immature weeks, with its correlation to the true label, as the leading indicator; never presented on the same axis as the true number.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The label window is known and stable, so "mature" is well-defined and the label-lag curve has the shape it had last quarter.
- The outcome join covers a stable, high share of predictions; a fall in join rate is a data incident that masquerades as improving quality.
- The proxy's correlation with the true label holds for the current population and model, which is only checkable retrospectively.
- Offline: from historical outcomes, plot the arrival curve of labels by days since prediction to find the real maturation window, including its tail.
- Online: the label-lag dashboard, daily; a join-rate monitor; proxy quality compared with the true quality for the weeks where both exist.
- Over time: after each retrain, a scheduled re-evaluation when its first predictions mature, with the promotion decision formally revisited.
What can go wrong
- The label window is "ninety days" in the design and, in the data, a long tail of chargebacks arrives at day 120; the mature threshold is set at 90 and a fraction of positives are permanently miscounted.
- The proxy is manual-review verdicts, which exist only for transactions the model flagged; proxy quality measures precision and says nothing about recall, and the team reads it as overall quality.
- The join key is the transaction id, and a refund creates a new transaction id that the chargeback lands on; a share of positives never join and the model looks better than it is.
- Reporting only mature quality means the dashboard's most recent real number is a label-window old, which is honest and uncomfortable.
- Proxies add pipelines, and each needs its correlation monitored, which is a monitor on a monitor.
- A promotion decision that is revisited when labels mature means a model can be demoted a quarter after it shipped, and the rollout process has to allow that.
- "Precision is trending down over the last eight weeks." Over the last eight weeks the label window has been closing. Check the label-lag curve before the model.
- "The new model's precision is fine." On which labels? If it was retrained six weeks ago and the window is ninety days, the fine number is the old model's, or it is computed on immature examples.
- "Use the proxy as the metric; it is faster." The proxy is faster and is a different quantity. It goes on the dashboard with its correlation attached, and the promotion decision waits for the true label or is explicitly provisional.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICA click model has labels in seconds and this lesson barely applies; a chargeback model waits ninety days, a default model a year, a treatment-outcome model longer. The architecture is the same; whether it is the central problem depends on the window.
- SIMULATEDThe three-week label delay and the null accuracy for the last three weeks of every scenario are the Drift Explorer's design, chosen to make the delay visible on a twelve-week run; real windows and their tails are longer and less regular.
Where the depth lives
This domain teaches the model and hands the rest off by name.