ML Observability
Tracing one request through features, model version, prediction, decision and outcome; logging what is necessary and safe; and debugging an incident from the business metric down.
A healthy model server can serve a wrong model indefinitely. Observability for a model means tracing Request → Features → Model Version → Prediction → Decision → Outcome, and watching signals service health does not have.
The prediction log is the monitor's input, the incident's evidence, the outcome join's left side and the next training set. Log what is necessary, reference what is sensitive, and decide retention before the first row.
One request id, from the click in the frontend through the backend, the feature service, the model server and the decision, to the outcome event weeks later. Each hop records something specific, and a hop that drops the id is where the next incident becomes a guess.
Conversion fell after a deploy. Investigate from the business metric down — prediction distribution, model version, feature values, feature pipeline, raw data — in that order, and do not retrain until the cause has a name.
A model incident postmortem records the assumption that broke, the signal that should have fired, the label delay that hid it and the test now added — and never concludes that "the model" was at fault.