Model Monitoring
A model needs everything a service needs, plus three distributions a service does not have: features in, predictions out, and outcomes back. Four layers, each with an owner, each catching a different failure.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The service is up, latency is fine, and nobody knows whether the model is still right. What does a deployed model need watched, and who watches each part?
A churn model has been in production for five months. The backend team's dashboard is green. The retention team says the call list "feels off" — they cannot say how — and nobody can point at a chart that would have shown it.
The model is a service. Monitor it like one: uptime, error rate, p99 latency, CPU. If those are green the model is working, and if the model were wrong the business would tell us.
The business did tell you — "feels off" — five months in, with no chart. A model returning confident scores at low latency from a broken feature pipeline is a healthy service and a broken model.
- The business did tell you — "feels off" — five months in, with no chart. A model returning confident scores at low latency from a broken feature pipeline is a healthy service and a broken model.
- Service metrics see the request; they cannot see that the feature vector's null rate tripled last Tuesday or that the score distribution shifted a week ago.
- Outcome quality — the only signal that says whether the predictions were right — needs a join between predictions and labels that was never built, and the labels are thirty days late anyway.
- The model version was bumped twice by a retraining job and nobody can say which version produced which week's list.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict thirty-day churn per subscriber; the label arrives when the thirty days end. The surrounding system's target is that the weekly call list continues to contain the customers most likely to cancel.
- Monitoring's own target is that any failure of that system is visible on a chart, with enough lead time to act, to a person whose job it is to look.
- The serving path logs request id, model version, feature vector and score. The outcome table records cancellations by subscriber and date. Nothing joins them today.
- Service metrics — request rate, error rate, latency — exist in the platform's standard dashboards. Feature and prediction distributions exist nowhere.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A deployed model fails at four layers. System: the service is down, slow or erroring. Data: the inputs arrive with a different distribution, more nulls, a changed schema. Model: the output distribution moves, or the model's quality against outcomes falls. Business: the metric the model exists to move stops moving.
- Each layer has a signal, a latency and an owner. System signals are immediate and belong to the platform. Data signals are available on the day and belong to whoever owns the feature pipeline. Model signals split — prediction distribution is immediate, quality waits for labels. Business signals are slowest and belong to product.
- The layers do not substitute for one another. A green system layer says nothing about the data layer, and a stable prediction distribution says nothing about quality once the world changes underneath it (Concept Drift).
Four layers, four owners
The system layer is the one the platform already gives you. The other three are the ones the model needs and nobody builds by default. Each catches a failure the others cannot: a null-rate spike is a data-layer event that the system layer calls healthy; a concept change is a model-layer event that the data layer calls stable; a product change that makes the prediction irrelevant is a business-layer event that every other layer calls fine.
The owner column is not decoration. An alert with no owner is an alert that will be muted, and the data layer in particular tends to fall between the feature-pipeline team and the model team.
| Layer | What it tracks | Latency of the signal | Owner | Catches |
|---|---|---|---|---|
| System | Request rate, error rate, latency percentiles, CPU/GPU, cost per request | Seconds | Platform / backend | Outages, saturation, a bad deploy |
| Data | Per-feature null rate, distribution distance to training reference, schema, freshness | Same day | Feature pipeline owner | Pipeline bugs, upstream changes, train/serve skew, input drift |
| Model | Prediction mean and positive rate (immediate); quality per version on joined outcomes (delayed) | Same day / label delay | Model team | Rollout skew, prediction drift, concept drift, decay |
| Business | The metric the model exists to move — retained revenue, review-queue precision | Weeks | Product | The model being right about something that no longer matters |
What the stable scenario teaches
The Drift Explorer's baseline scenario is twelve weeks in which nothing changes. It exists so the learner sees what the monitors look like when the world is still: PSI is never exactly zero on four hundred rows a week, prediction mean wobbles, and the last three weeks of accuracy are blank because their labels have not arrived. A team that has not looked at the stable case will alert on noise and read the blank weeks as a gap.
The thresholds on the page — PSI above 0.2, null rate above 5%, prediction mean moving more than 0.1, quality more than 0.05 below week 0 — are the explorer's, chosen for a synthetic model. The mechanism is the lesson; the constants are not.
1def weekly_readings(log, reference, week, label_delay, today):2 rows = [r for r in log if r.week == week]3 served = [r.features["x0"] for r in rows]4 reading = {5 "week": week,6 "model_version": most_common(r.version for r in rows),7 "null_rate": mean(r.features["x0"] is None for r in rows),8 "psi_x0": psi(reference["x0"], [v for v in served if v is not None]),9 "prediction_mean": mean(r.score for r in rows),10 "positive_rate": mean(r.score >= 0.5 for r in rows),11 "accuracy": None, # not yet — the labels are not here12 }13 if week + label_delay <= today:14 joined = [(r.score >= 0.5, r.outcome) for r in rows if r.outcome is not None]15 reading["accuracy"] = mean(pred == y for pred, y in joined)16 reading["join_rate"] = len(joined) / len(rows)17 return readingThe accuracy field is None on purpose for recent weeks. A dashboard that fills it with the last known value is lying about the present; one that leaves it blank is telling you the truth about the label delay. The join rate is the monitor on the monitor.
The assumption every other monitor rests on
Every monitor in the model and data layers is computed from the prediction log. If the log is incomplete, the monitors measure a sample; if it lacks the version, readings from two models blend; if the join key is wrong, quality is measured on whoever happened to match. The log is the assumption underneath the assumptions.
It should be treated as production data with its own freshness and completeness checks — the Data Engineering domain's tooling applies to it directly.
Every served prediction is logged with model version, feature vector and a key on which the outcome will later join, and the log is complete enough that monitors computed from it describe production.
holds when Logging is on the serving path rather than best-effort; the log volume matches the request count; the outcome join rate is high and stable; the version is recorded per request, not read from the registry at query time.
breaks when A logging sink backs up and drops rows under load — exactly when traffic is unusual; the outcome table deletes churned subscribers; a rollout serves two versions and the log records one.
respond Treat a logging gap as a monitoring outage: the readings for that window are unknown, not fine, and the dashboard should show them as unknown.
How to build it
Most important first.
- Log every prediction with its model version, feature vector and a join key, so that outcomes can be attached later and every other monitor can be computed from the log (Prediction Logging).
- Build the four layers as four dashboards with four owners, and write down the owner. A data-layer alert that pages the backend team gets muted.
- Track feature distributions per feature against a training-set reference, prediction distribution against the validation-set reference, and quality per model version as labels arrive (Data Drift, Prediction Drift, Ground-Truth Delay).
- Add cost and version to every chart: cost because a model that quietly doubled its inference spend is a failure, version because every reading needs to be attributable to the artifact that produced it (Inference Cost, The Model Registry).
- Alert on symptoms with a documented response, not on every distance metric crossing a line (Drift Is Not Failure).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-layer: request error rate and latency percentiles; per-feature null rate and distribution distance; prediction mean and positive rate; quality at the operating threshold per week, per model version; the business metric per week.
- The number that maps to the decision "is the model still right" is quality on joined outcomes, and it is the last one to arrive. Every other number is a leading indicator whose job is to say where to look.
- Do not report the validation metric from training time on the production dashboard as if it were current. It is a fact about the past.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every prediction is logged with the model version and a key that the outcome can be joined on; a logging failure is a monitoring outage, and is itself monitored.
- The reference distributions the monitors compare against are the ones the current model was trained on, and are updated when the model is.
- Each of the four layers has a named owner who receives its alerts and has a documented response.
- Offline: before deployment, compute every monitor on the validation set to establish its reference and its natural variance, so alert thresholds are set from data rather than from a default.
- Online: on rollout day, confirm each layer is reporting for the new version — a model can serve for a week with the feature monitor pointed at the old reference.
- Over time: a monthly check that the outcome join still covers most predictions; a falling join rate means the quality metric is silently measuring a subset.
What can go wrong
- Every feature drifts a little every day, the data-layer dashboard is permanently red, and the one real signal is lost in it (Alert Fatigue: The Page Nobody Reads in the Observability domain).
- The join between predictions and outcomes is built on subscriber id, and subscribers who churn are deleted from the outcome table, so churners vanish from the quality metric.
- The version label on the dashboard is the version the registry says is current, not the version each request was served by, and a slow rollout makes every reading a blend.
- Logging every feature vector is storage, a privacy question (ML Privacy) and a join to maintain.
- Four dashboards with four owners is organisational work, and the owners will disagree about whose layer a given incident belongs to.
- Per-feature monitors scale with the feature count, and each one is a threshold to tune.
- "The service is healthy, so the model is fine." The service layer is one of four. A model returning confident nonsense is a healthy service.
- "We monitor accuracy." Then the dashboard shows accuracy for weeks whose labels arrived, which is the weeks before the problem started.
- "MLOps means Kubernetes, so once we are on it monitoring is handled." The platform gives the system layer. The other three are the model team's to build.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe four layers apply to every deployed model; what differs by task is the model-layer signal — positive rate for a classifier, mean and spread for a regressor, top-k overlap for a ranker.
- SCALE-SPECIFICA single batch model scored weekly can be monitored with one notebook and one owner; at dozens of online models the per-feature monitors and the outcome join need a platform, and the ownership split becomes the hard part.
- SIMULATEDThe readings in the Drift Explorer — PSI, null rate, prediction mean, delayed accuracy — are computed by a synthetic model on generated traffic, for the shape of the argument; no number here is a measurement of a real system.
Where the depth lives
This domain teaches the model and hands the rest off by name.