Target Definition
The target must encode the outcome you actually care about, at a horizon, from a moment. churned = cancelled within 30 days of the snapshot is a target; churned is not.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What makes a target definition precise enough to build a dataset from, and how does an imprecise one produce a model that is correct about the wrong thing?
The analytics lead: "Our churn model predicts churn well, but finance says churn is going up while the model says it is flat. Whose churn is right?"
There is a cancelled_at column. Churn is when it is set. The label is one line of SQL and it is obviously right.
The column captures the click, not the outcome. Annual subscribers who lapse without clicking are negatives in the training data and churners to finance; the model learned that annual subscribers never churn.
- The column captures the click, not the outcome. Annual subscribers who lapse without clicking are negatives in the training data and churners to finance; the model learned that annual subscribers never churn.
- Downgrades from the premium plan are invisible to the label and most of finance's lost revenue; the model is silent on the outcome the business cares about.
- The label has no horizon. A subscriber who cancels three years after the snapshot is a positive in the same class as one who cancels next week, so the model predicts "will eventually leave", which is everyone, weakly.
- The label has no moment. The dataset was built once, so a subscriber's label depends on when the query ran, and rebuilding it a month later changes the labels of the last cohort.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The model's target, as written in the training query, was
cancelled_at IS NOT NULLat the time the dataset was built. Finance's churn is revenue lost from subscribers who did not renew at the end of their term, including downgrades. - Neither is wrong. They are different outcomes, and the model was built to predict one of them without anyone writing down which.
- Subscription records with
started_at,plan,cancelled_at,term_end, and a plan-change history; billing events; the monthly snapshots the model trains on. - A subscriber who cancels sets
cancelled_atbut keeps access untilterm_end; a subscriber who downgrades sets nocancelled_atat all; an annual subscriber who does not renew setscancelled_atonly if they click, and many simply lapse.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A target is the answer to four questions: what event, defined as a rule over raw records; for which population, defined by a state at the snapshot; from what moment, the snapshot timestamp; to what horizon, the window in which the event counts. Remove any one and the label is under-specified — it will still compute, and it will compute something other than the outcome the decision cares about.
- The event rule is where the business meaning lives. "Cancelled within 30 days" and "did not renew, or downgraded, within 30 days of term end" are different outcomes with different base rates, different features that predict them and different actions that address them. The model will learn whichever one the rule encodes, faithfully.
- Horizon and moment together define the label as a function of the snapshot: label(user, t) = event occurs in (t, t + horizon]. This makes the label stable — recomputable at any later time with the same answer — and makes the prediction moment explicit, which is what keeps future information out of the features (Label Leakage).
Raw events → business rule → label
The label is not a column. It is the output of a rule applied to raw events, at a moment, over a horizon, and the rule is where the business meaning is encoded. Drawing it as a pipeline makes the point that each arrow is a place the meaning can be lost: the events may be incomplete, the rule may encode the click rather than the outcome, the moment may be missing.
The rule belongs in versioned, tested code with the same status as the model. A change to it is a change to what the model predicts, and every artifact trained on the old rule is a different model from one trained on the new.
Four definitions, four models
Each row below is a plausible reading of "churn" and each produces a different dataset, base rate and model. The naive definition is the first row. The one the retention decision needs is the last. The offline number for the first is the best and its production value is the worst, because it is the easiest to predict and the furthest from the outcome.
The table is worth keeping for every target you write: enumerate the definitions before choosing one, and record why the others were rejected.
| Definition | Event rule | Horizon | What it misses | Who it serves |
|---|---|---|---|---|
cancelled_at IS NOT NULL | The cancellation click, ever | None | Lapses, downgrades; mixes cohorts across years | Nobody — it is a column, not an outcome |
| Cancel click within 30 days | Click in (t, t+30d] | 30 days | Lapses and downgrades; annual plans look immortal | A monthly-plan retention list |
| Any end of paid access within 30 days | Click, or lapse at term end, in (t, t+30d] | 30 days | Downgrades — the largest revenue loss | The retention team, mostly |
| Revenue loss within 30 days | Click, lapse, or downgrade in (t, t+30d], weighted by revenue | 30 days | Nothing finance counts; harder to predict | The retention team and finance, with one number |
The target that drifted away from the business
A target that was right at launch can become wrong without any change to the data or the model. The product adds a pause feature; billing moves to annual; finance stops counting involuntary payment failures. Each changes what "churn" means to the decision while the rule in the label pipeline keeps computing what it always computed.
This is a formulation assumption rather than a data one, and the monitor for it is a reconciliation rather than a distribution check: does the label's aggregate still agree with the number the business trusts?
The event rule, population rule, moment and horizon still describe the outcome the retention decision is trying to prevent, as the business currently defines it.
holds when The monthly label base rate reconciles with finance's churn figure within tolerance, and no product or billing change has introduced a subscription state the rule does not handle.
breaks when A pause feature, a new plan tier, a billing-term change or a redefinition of churn by finance changes the outcome without changing the rule.
respond Reopen the target definition with the decision owner and finance, version the new rule, rebuild the labels and retrain — a target change is never a retrain alone.
How to build it
Most important first.
- Write the target as a sentence with all four parts, agreed with the decision owner and with finance, before writing the SQL: "for subscribers active on the snapshot date, a cancellation click, a lapse at term end, or a downgrade, occurring within 30 days of the snapshot" (Problem Formulation).
- Derive the horizon from the decision — the retention team acts within a month — not from convenience, and record why (Decision Before Model).
- Express the rule as versioned, tested code over raw events, with the snapshot timestamp as an input, so the label for any (user, t) is reproducible (Label Construction).
- Reconcile the label's aggregate rate against the business's own churn number each month; a divergence means the target and the business have drifted apart (Business Metrics vs Model Metrics).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The label's base rate per month and per plan, reconciled against finance's churn figure computed from the same definition. Agreement is the evidence the target means what the business means.
- The model's quality on the target it was defined for, on a time-based split at the snapshot moment. This is the number that says the model predicts the outcome; it says nothing if the outcome is the wrong one.
- Do not measure the model's quality against a different definition of churn than the one it was trained on and call the gap a model problem.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The event rule still captures the outcome the business means by churn — cancellation, lapse and downgrade, with the same treatment of pauses, payment failures and plan changes.
- The thirty-day horizon still matches the window in which the retention action can change the outcome, and the billing terms still put enough events inside it.
- The population rule — active on the snapshot date — still describes the subscribers the decision applies to.
- The label's monthly base rate still reconciles with the business's churn figure within a stated tolerance.
- Offline: unit tests over the label rule with synthetic event histories — a lapse, a downgrade, a pause, a cancel-then-reactivate — asserting the label for each at several snapshot dates.
- Online: monthly reconciliation of the label base rate against finance; an alert when the two diverge by more than the agreed tolerance.
- Over time: a review of the target sentence whenever the product changes how subscriptions can end — a new pause, a new plan tier, a new billing term.
What can go wrong
- The four-part target is agreed and encoded, and the product introduces a pause feature; paused subscribers are neither cancelled nor active, and the rule silently treats them as negatives.
- The horizon is thirty days and the billing term changes to annual for most subscribers; the event now clusters at term end and the thirty-day label is empty for eleven months of each year.
- The label is reconciled at launch and the business quietly redefines churn to exclude involuntary payment failures; the model keeps predicting the old definition and its "churn" diverges from the dashboard again.
- A target that includes lapses and downgrades has a later, messier label than a cancellation click — lapses are known only at term end — so the training data is further behind the present.
- Agreeing the definition with finance and the retention team is a negotiation, and the model waits for it.
- A composite event — click or lapse or downgrade — is harder to predict than any one of them, and the offline number for the honest target is lower than for the convenient one.
- "There is a churn column, so the target is defined." The column records one mechanism by which a subscription ends. The target is the outcome the decision cares about, and it usually spans several mechanisms and excludes some the column includes.
- "The model predicts churn well, so finance's number is wrong." Both numbers are right about their own definitions. The model was never pointed at finance's.
- "Pick a horizon later, once we see what works." The horizon decides which rows are positive, so it decides the dataset; trying several is a form of tuning on the label, and the one that "works" best offline is often the one that leaks the most.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALEvent, population, moment and horizon are required for any supervised target; for regression the event becomes a quantity measured at the horizon, and for ranking it becomes an ordering, but the four questions do not change.
- DOMAIN-SPECIFICSubscription churn has a recordable event; in credit risk the event is a default defined by a regulatory rule, in medicine a diagnosis that may itself be wrong, and in fraud a chargeback that only arrives if someone complains — the same four questions have much harder answers there.
Where the depth lives
This domain teaches the model and hands the rest off by name.