Designing a Churn Prediction System
A weekly call list for a team of fixed capacity. Batch scoring, a threshold that is a queue size, labels a month late, explanations the callers can use — and a demonstration of why the online endpoint someone will propose is unnecessary.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How is a churn prediction system designed around the decision it serves, and why is online inference the wrong default for it?
Accounts are cancelling and we find out when they do. The customer-success team can have about two hundred conversations a week and wants to spend them on the accounts most likely to leave — ideally a week before renewal, with a reason they can open the call with.
Export a table with one row per account, train a classifier with a random split, deploy it behind an endpoint so any surface can ask "is this account at risk" in real time, and report validation AUC weekly. The number is excellent.
The random split put adjacent weeks of the same account on both sides; validation measured memorisation. Re-split by time and account, and the gap over the logistic baseline shrinks to something that has to be earned (Entity Leakage).
- The random split put adjacent weeks of the same account on both sides; validation measured memorisation. Re-split by time and account, and the gap over the logistic baseline shrinks to something that has to be earned (Entity Leakage).
- The subscriptions join pulled in
updated_atand a nulled renewal date, both written at cancellation; the model was reading the answer through two columns that look nothing likecancelled_at(Temporal Leakage). - The endpoint was built and a dashboard banner started calling it on page load. The model added 80 ms to a request nobody had a budget for — and the score it returned was already sitting in Monday's table.
- Three months in the list "became useless": an onboarding overhaul halved churn prevalence, the top 200 contained half as many churners at the same ranking quality, and the rule baseline fell with it. Nothing about the model had changed (Accuracy Under Imbalance).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an active account will initiate a cancellation with an effective date inside the thirty days after a weekly snapshot. Involuntary churn from failed payments is a different event with a different fix and is excluded from the positive class (Target Definition, Label Construction).
- The decision is a ranked list of roughly two hundred accounts per week, with a reason per account. The model output the team consumes is the list, not a score, and the threshold is the two-hundredth account's rank (Prediction vs Decision).
- One example is one account at one Monday snapshot, with features computed from data timestamped before that Monday — activity over 7, 30 and 90 days aggregated from users, seat utilisation, support tickets as they were at creation, payments with retries collapsed, plan and tenure — and the label from the thirty days after. An account contributes one correlated row per week for as long as it is active (What Is One Example?).
- The subscriptions table is the label source and shows current state, not history; every honest feature from it requires a point-in-time reconstruction from an audit log or daily snapshot (Point-in-Time Correctness).
- Once the model is live, the retention call is an intervention that changes the label; it is recorded as a treatment column, and a random holdout of flagged accounts receives no call so that the label still means what it says (Feedback Loops).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The decision is consumed on Monday by a team working a list. The prediction is needed a week before renewal, not at page load. The features are weekly aggregates that do not change meaningfully within a day. The population is every eligible account, which is small. Each of those facts points at batch, and together they make online inference a design that would add a model server, a feature cache with its own staleness, a latency budget and an on-call rotation to deliver a score at a moment nobody is waiting for it (Batch Inference, Choosing the Inference Mode).
- A capacity threshold turns the model into a ranker in practice. Precision in the top 200 is the metric the rule baseline and the model can share, and revenue weighting — probability times annual value times an assumed save rate — is a decision layer on top of the score, versioned and tested separately (Business Metrics vs Model Metrics). It also means precision moves when prevalence moves, and that has to be monitored as a base-rate change, not model decay.
- The label needs thirty days to exist, so the last labelled snapshot is always a month old, every outcome metric lags a month, and a "precision this week" panel computes on the handful of fastest cancellations — a different quantity from what its title claims (Ground-Truth Delay).
- The callers need a reason. A per-account explanation — the features that moved this account's score most — is a requirement of the decision, not a nice-to-have, and it constrains the model family or adds an attribution step whose approximations must be stated (Explainability, Attribution Is Not Causality).
The system, drawn around a Monday
The architecture is a scheduled job, a warehouse query and a table, and that is its strength. Every box in the general picture (ML System Design Architecture) is present — the registry, the contract test, the prediction log with versions, the monitors — but serving is a job that writes a table joined by the CSM tooling, and "rollout" is running two models on the same week.
The one online consumer in the diagram is the in-app banner that the product team will eventually build. It reads the same table. The capstone's +80 ms injection is what happens when it calls a model instead.
Why the online endpoint is unnecessary
The proposal will arrive: "expose the model as an endpoint so any surface can ask". It sounds like flexibility. What it adds is a second serving path with its own feature source, its own freshness, its own latency budget and its own pager, to compute at request time a score that changes weekly and already exists. The comparison below is the argument in one place; the capstone's latency injection is what it looks like when it is lost.
The honest exception is a decision that genuinely needs request-time context — an offer shown in the cancellation flow, triggered by the click on "cancel". That is a different target, a different horizon, different costs and a different design. It should be formulated separately rather than served by stretching this one (The Questions Before the Boxes).
| Option | Quality | Latency | Cost | Operational | Note |
|---|---|---|---|---|---|
| Weekly batch + lookup | Freshness bounded by a week, which the decision tolerates; one path, one set of monitors. | ||||
| Online per request | Same score quality with weekly features; the latency and the pager are paid on every request whether or not anyone needed freshness. | ||||
| Hybrid (batch score + live signal) | Only justified if a request-time signal actually moves the decision; then it is a second formulation. |
caveat The scores assume the decision is the weekly list. For a cancellation-flow offer the quality column would favour the online option because request-time context carries real signal there — which is the point: the matrix is a property of the decision, not of the model.
Feature lookup (cache with cold-miss to the warehouse) + transformation + network hop + model, per dashboard load, with a fallback, a latency SLO and an on-call rotation — to return a score that was computed from weekly aggregates.
The weekly job writes a versioned score per account; the banner reads it with a keyed lookup; the threshold review, the shadow comparison and every monitor already exist once and cover both consumers.
The decision does not need a fresher score than the features can provide, and the features change weekly. Online inference buys freshness the inputs do not have, at the cost of a second path that can skew. Latency is a job duration here, which makes a slow model a cost problem rather than a user-facing one.
Eight things that go wrong, and none is fixed by retraining
The capstone at /ml/capstone injects eight failures into exactly this design, in the order a real project meets them. Each presents as something the model did wrong, each has a retrain or a model swap that looks like the fix, and in each the mechanism is somewhere else. The table names them so that the pattern is visible before the exercise: the offline number, the label, the pipeline, the serving path, the decision consumer.
What the eight have in common is that the correct response starts by reading the monitors by slice and naming the mechanism. The retrain reflex is expensive not because it wastes a training run but because it launders the failure into the weights, after which the monitors go quiet.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Random split over weekly rows | Validation number far above the baseline | Same account in train and test — memorisation measured as generalisation | Time-and-account split with a gap; re-evaluate everything (Entity Leakage) |
| Subscription columns added | Top-200 precision nearly doubles overnight | updated_at and nulled renewal written at cancellation — future timestamps as features | Point-in-time reconstruction; leakage audit by feature group (Temporal Leakage) |
| Onboarding overhaul | Precision halves; features and AUC stable | Prevalence changed; the queue holds fewer true positives at the same ranking quality | Queue-size conversation; add prevalence to the headline (Accuracy Under Imbalance) |
| Aggregation job fails on the weekend | List is all iOS-heavy accounts | Stale features served as current; stale looks like inactive | Freshness gate on the scoring job; stale imputed as missing (Feature Freshness) |
| Neural challenger in shadow | List overlap under 20 of 200 | Serving normaliser differs from the training one | Ship the scaler in the artifact; contract test in CI (Train / Serve Skew) |
| Banner calls a new endpoint | +80 ms; dashboard p95 SLO breached | Feature fetch, hop and transform — not the model — for a score that already exists | Read the weekly table; measure the breakdown first (Latency Breakdown) |
| Support tool migration | Score distribution shifts; top 200 all EMEA | Ticket sync broke; feature drift with a data cause | Repair the sync, re-score, record the incident (Drift Is Not Failure) |
| Leadership asks on Wednesday | "Precision this week" swings wildly | Labels are 30 days late; the panel measures the fastest churners | Age-labelled metrics; proxies labelled as proxies (Ground-Truth Delay) |
How to build it
Most important first.
- Batch, weekly, from the warehouse with a point-in-time query parameterised by snapshot date; the same SQL builds training rows as of any past Monday and scoring rows as of this one, which removes the most common skew source by construction.
- Time-and-account split with a thirty-day gap at each boundary; two baselines measured the way the model will be — the team's current rule and a regularised logistic regression on the same features — with "precision in the top 200 per week" as the shared number (Time-Based Split, Beating the Baseline).
- Fitted preprocessing shipped inside the artifact and executed by the same code path in the weekly job; a serving contract test that scores stored rows and compares to values recorded at training, run in CI (Preprocessing Lives in the Artifact, Serving Contract Tests).
- Champion/challenger on the same weeks: the challenger runs in shadow for several cycles, lists are compared on overlap and on labelled weeks, then capacity is split between the lists before promotion. A promotion takes at least six weeks end to end because of the label lag, and that is the design, not a delay (Champion / Challenger).
- Monitoring in the order things can fire: feature distributions and freshness per slice today; score distribution and top-200 composition today; intervention rate and holdout size today; precision at 200 against the baseline, thirty days late, on the holdout.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision in the top 200 per week, revenue-weighted, on the holdout population, against the rule baseline on the same weeks — the only number that maps to two hundred calls and lost revenue.
- Weekly prevalence beside it, because a fall in precision with stable ranking quality is a base-rate change and the response is a business conversation about queue size, not a retrain.
- AUC as a diagnostic of ranking quality across the population; calibration on validation weeks if the CSM lead reads scores as probabilities; accuracy nowhere, since predicting "no" for everyone scores in the high nineties and calls nobody.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The weekly feature tables are fresh to within their expected lag when the scoring job runs, and a stale slice is detected before the list is released.
- The retention team's capacity is still roughly two hundred conversations a week, and the threshold — the 200th rank — is reviewed when it changes.
- A random holdout of flagged accounts continues to receive no intervention, so the outcome metric has a population on which the label still means cancellation rather than "cancellation despite a call".
- The label definition — customer-initiated cancellation, thirty-day horizon, involuntary churn excluded — has not been changed by a billing-system migration without the dataset version changing with it.
- Offline: re-materialise the training set from the dataset version and check prevalence and row counts; run the leakage audit by feature group; confirm the validation number was produced by the time-and-account split with the gap.
- Online: shadow-compare the challenger's weekly list to the champion's on overlap and composition before anyone acts on it; check feature freshness and slice row counts before every release of the list.
- Over time: at thirty days and older, precision at 200 against the rule baseline on the holdout; weekly prevalence beside it; holdout size as a monitored quantity with an alert when it shrinks.
What can go wrong
- The mobile aggregation job fails over a weekend and Monday's list is all iOS-heavy accounts: stale features look exactly like inactivity. The scoring job needs a freshness check that refuses to run on old feature tables and imputes stale as *missing*, not zero (Feature Freshness).
- A neural challenger with a hand-re-implemented scaler in the scoring job ranks accounts nearly inverted relative to the champion; the weights are right and the serving normaliser is wrong (Train / Serve Skew).
- A regional support-tool migration stops syncing EMEA tickets;
days_since_last_ticketjumps, the top 200 turns EMEA, and a colleague proposes an emergency retrain "on the drifted data" that would launder the artefact into the weights (Drift Is Not Failure). - The holdout is eroded by a well-meaning manager who does not want to leave at-risk accounts uncalled; after two quarters the outcome metric is computed on a population the team acted on and means nothing.
- Weekly batch means an account that starts disengaging on Tuesday is not on a list until the following Monday; the design accepts that because the intervention is a call scheduled days out anyway.
- The holdout is a set of at-risk accounts deliberately not called so that the effect of calling can be measured — a real cost to real revenue, argued about every quarter.
- A capacity threshold makes the score scale irrelevant across versions, which removes a rollout hazard, and makes precision sensitive to prevalence, which adds a monitoring burden.
- "We need an online endpoint so the product can show risk in real time." The account's weekly score already exists in a table; a lookup takes a fraction of a millisecond and reuses the threshold review and monitoring the batch system has. If a genuinely fresh signal is needed for a banner, that is a different decision with its own design.
- "Precision fell, retrain." Check prevalence first. If ranking quality on recent labelled weeks is unchanged and the rule baseline fell too, the base rate moved and the answer is a queue-size conversation.
- "The neural model does not generalise; go back to trees." The neural model received inputs scaled differently from training. Trees survived the same pipeline only because they are indifferent to monotone scaling, which is a property of the family, not evidence about the pipeline.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICThe weekly-list shape is specific to a decision consumed by a human team with fixed capacity; a churn model that drives an automated in-app offer at session time is a different decision with a different latency and a different threshold, and would be designed separately.
- SIMPLIFIEDThe 200-call capacity, the 80 ms latency figure, the thirty-day horizon and any percentages are illustrative values chosen to make the argument concrete; a real product substitutes its own and the reasoning is unchanged.
- CONTESTEDSerious practitioners argue for a fixed monthly retraining cadence over a triggered one: it keeps the pipeline exercised, bounds staleness, and removes the judgement call that "retrain on evidence" requires at 2 a.m. The cost they accept is retraining on intervention-contaminated data whenever the holdout has been neglected, which a triggered pipeline with a gate at least has to confront explicitly.
Where the depth lives
This domain teaches the model and hands the rest off by name.