Offline metrics are strong but production quality is poor. What do you investigate?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A demand forecasting model for a grocery retailer was validated on the last twelve weeks of history with a healthy error reduction against the incumbent (illustrative). Three weeks after deployment, store managers report over-ordering of fresh produce and the waste metric has gone up. The offline evaluation, re-run today on the same validation set, still shows the same strong numbers.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Incident channel excerpt (illustrative): ops: waste on fresh produce +9% vs last month, worst in west-coast stores ds: re-ran validation on the same 12-week set: MAE unchanged, still 14% better than incumbent ds: model weights match registry sha, serving container unchanged since deploy eng: serving pulls sales_yesterday from the ops DB (local store time), training used the warehouse daily table (UTC) eng: promo_flag is null for new-format stores; training pipeline dropped nulls, serving casts to 0
What it is really testing
Whether the candidate has an ordered list of ways the same weights can behave differently in production, and whether they know that "the offline number is still good" is not reassuring — it is the symptom. The best candidates start from the business metric and trace backwards through the serving path.