The 97% Accurate Churn Model
Decide what you would do from the brief alone, including whether you would change anything at all. Everything below it is available, but the exercise stops working if you open it first.
Slack from the analytics lead: "Great news, the churn classifier is at 97% accuracy on the test set. Marketing wants to start the win-back campaign on everyone it flags next Monday. Can you confirm it is ready?"
Confirming readiness because the accuracy is high and the test set was held out properly. Everything about the process was correct — the split, the holdout, the reporting — and it produced a number that cannot be wrong and cannot be useful. The campaign runs on Monday, contacts almost nobody, and marketing concludes churn modelling does not work. The fix that actually looks like work — "rebalance the classes and retrain to push accuracy higher" — makes it worse, because a reweighted model flags thousands of customers with no change in how many of them would have left.
Read this even if you are confident. It is here rather than behind a button because it is the answer most teams actually ship, it passes review, and its cost arrives weeks later when the labels do.