A feature distribution changed. Should we retrain immediately?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A credit-risk team has a drift monitor on every input feature. On Monday morning it fires on device_type: the share of requests from a new mobile app version jumped from near zero to 40% over the weekend after a release (illustrative). The on-call engineer's runbook says "drift detected: trigger retraining". Decisions from this model take 60 to 90 days to resolve into a label.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Drift monitor — Monday 07:10 (illustrative) feature statistic baseline current status device_type PSI — 0.41 ALERT income_declared PSI — 0.02 ok age_bucket PSI — 0.01 ok prediction_p PSI — 0.05 ok approval_rate — 61.2% 61.8% ok Note: device_type value "mobile_v5" not in training vocabulary → encoded as UNKNOWN (12% of rows). Last resolved label cohort: applications from 11 weeks ago.
What it is really testing
Whether the candidate knows that drift is a signal, not a verdict. The strongest answers distinguish feature drift from prediction drift from concept drift, ask what happened to the *output* and to *quality*, and know that with a 60–90 day label delay, retraining today trains on the old world anyway.