A model unexpectedly reaches 99.9% validation accuracy. What do you investigate?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A fraud team has spent three months on a transaction classifier that hovered around a validation AUC in the low 0.8s (illustrative). A new engineer joins, adds a handful of features from the operational warehouse, and the next morning validation accuracy is 99.9% and AUC is 0.999. The engineer has opened a pull request titled "new features — huge win" and the product manager wants it in production this week.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Pull request #412 — "new features — huge win" Validation (random 80/20 split, stratified on label): accuracy 0.999 (was 0.962) ROC AUC 0.999 (was 0.83) PR AUC 0.97 (was 0.31) (numbers illustrative) New features added from warehouse table `txn_enriched`: amount_zscore_30d rolling z-score of amount, customer, 30 days merchant_fraud_rate_90d fraction of merchant's transactions flagged, 90 days customer_dispute_count count of disputes on the customer account days_since_last_chargeback days since most recent chargeback on the card hour_of_day, is_weekend Note from author: "customer_dispute_count and merchant_fraud_rate_90d are computed as of the snapshot date of the table, which is refreshed nightly."
What it is really testing
Whether the candidate treats a too-good number as a *symptom* rather than a result. The strongest signal is that they have a checklist of mechanisms, ordered by likelihood — leakage, split strategy, duplicated entities, target-derived features, plain data bugs — and that they can say how to confirm each one rather than just naming it.