What should you log for every prediction, and what must you not?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A health-insurance recommender logs the full request payload, including free-text notes, to a shared log cluster with 90-day retention so that "we can debug anything". Privacy has flagged it. Meanwhile, the on-call engineer cannot answer "why did member X get this recommendation on Tuesday" because the log has the input but not the model version, the feature vector or the threshold.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Current log line (illustrative, redacted)
{"ts":"...","member_id":"M-88213","dob":"...","notes":"patient reports ... [free text, 1.2 KB]",
"request":{...full payload...},"score":0.71,"recommended_plan":"P-12"}
Missing: model version, artifact hash, feature vector, threshold/decision-rule version, request id.
Retention: 90 days, cluster-wide read access.What it is really testing
Whether the candidate knows what a prediction log is *for* — reconstructing a decision, joining to outcomes, monitoring — and can therefore say what it needs, and whether they treat sensitive data as a design constraint rather than an afterthought: minimise, separate, hash, control access and retain deliberately.