Problem Formulation
"Users are cancelling subscriptions" is a situation, not a task. Six questions turn it into Input X → Model → Prediction ŷ → Decision, and each one skipped is a model that answers something nobody asked.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Someone says "users are cancelling subscriptions, can we predict it?" What has to be decided before that sentence becomes a modelling task, and in what order?
The head of growth: "Churn is up. We have all this usage data. Can't we build a model that tells us who is going to cancel so we can do something about it?"
The label is cancelled_at IS NOT NULL, the features are everything in the usage table, the split is random, the model is whatever scores best, and the output is a churn probability per user. Ship it to the retention team.
The probability arrives per user with no date attached; the retention team does not know whether a high score means "this week" or "eventually", so they cannot prioritise it against the two-week rule they already trust.
- The probability arrives per user with no date attached; the retention team does not know whether a high score means "this week" or "eventually", so they cannot prioritise it against the two-week rule they already trust.
- The label mixes users who cancelled years ago with users who cancelled yesterday, and the features are computed over each user's whole history — including usage after they decided to leave. Offline the model is superb; in production it flags users who have already gone.
- Nobody asked what the team would *do* with a score. The offer they can make costs a month of revenue, so a false positive costs real money, and the model was tuned for a metric that treats every mistake alike.
- Trial users, who cancel at ten times the rate of paying users, dominate the positives; the model learned "is on trial" and the retention team already knew that.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Not yet defined — that is the point. "Cancel" could mean the cancellation click, the end of the billing period, non-renewal after a trial, or a downgrade; "going to" could mean within a week or a year. The target is the output of formulation, not its input.
- By the end of the lesson: whether an active subscriber cancels within thirty days of a monthly snapshot, predicted on the first of the month, consumed by the retention team's weekly call list.
- Subscription records with start, plan, billing events and a
cancelled_attimestamp; product usage events per user per day; support tickets; a CRM log of retention calls made and offers given. - The retention team's current list is "anyone who has not logged in for two weeks", scanned by hand every Monday.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A formulation is a chain: a decision someone makes, an event whose prediction would improve that decision, a moment at which the prediction must exist for the decision to use it, an action that follows from the prediction, a cost for each kind of mistake, and a label that can be observed after the fact. Each link constrains the next, and the model is chosen last because everything about it — the target, the features that are legal, the metric, the threshold — is fixed by the links before it.
- The six questions, in order: what decision do we want to improve? what event should be predicted? when must the prediction be available? what action follows? how costly are false positives and false negatives? can labels be observed reliably? Only then: Input X → Model → Prediction ŷ → Decision.
- The order matters because the questions get more expensive to change as you go. Changing the decision after the target is built means a new dataset; changing the prediction moment after features are built means discarding half of them; discovering labels cannot be observed after training means discarding everything.
From a sentence to a task
The sentence "users are cancelling subscriptions" contains no target, no moment, no action and no cost. Formulation is the process of adding them, in the order that lets each one constrain the next. The worked answers below are for the running example; every other module in this domain assumes they were answered.
Note what the last row produces: not a model, but a specification a model can be built against and evaluated by. The model comes after.
- 1Decision
Which subscribers the retention team calls this week, given a fixed weekly capacity.
fails by A score nobody has a process to act on.
- 2Event
The subscriber cancels —
cancelled_atset — within the next thirty days.fails by A label with no horizon: "ever cancels".
- 3Moment
The first of each month, for every subscriber active that day, using only what is known then.
fails by Features from after the decision to leave.
- 4Action
A call and a one-month discount offer, for the top of the list up to capacity.
fails by An action nobody budgeted.
- 5Costs
Wasted offer: one month of revenue on a subscriber who would have stayed. Missed cancellation: the subscriber's remaining lifetime value.
fails by A metric that weighs them equally.
- 6Label observability
Cancellations are recorded for everyone within the horizon; contacted subscribers are marked so their outcome can be separated.
fails by The action erases the label it was meant to predict.
- 7Task
X = state on the first of the month → Model → ŷ = P(cancel within 30 days) → Decision = top-k by ŷ, k = weekly capacity.
fails by A model chosen before any of the above.
The number that measured the wrong question
The naive model was evaluated honestly on the dataset it was built from, and the dataset was built from a formulation nobody wrote down. The gap between its offline number and its production value is not a modelling failure. It is the difference between the question the dataset asked and the question the retention team needed answered.
This is the first place the domain's signature device appears, and the mechanism is the plainest one: the offline evaluation is only as meaningful as the formulation behind it.
Strong ranking quality on a random held-out set of all subscribers, labelled by whether cancelled_at was ever set.
The retention team stopped using the list after two weeks: most of the top-ranked users had already cancelled or were on trials the team never calls.
- 1No prediction moment: features were computed over each user's full history, so the model saw the usage collapse that follows the decision to leave, and ranked the already-gone highest.
- 2No horizon: "ever cancelled" includes users who left years ago, so the model learned what old cohorts looked like, not what next month's leavers look like.
- 3No decision: the model ranked all subscribers, but the team only calls paying users with a remaining term, and the ranking above the cut-off was mostly users outside that population.
What the formulation assumes will stay true
A formulation is itself a set of assumptions: that the decision exists, that the moment is right, that the action is available, that the costs are as priced, that the label is observable. The model inherits all of them, and none of them shows up in any metric, because the metric is computed inside the formulation.
The one that breaks most often is the first. Decisions move — teams reorganise, offers change, a product feature replaces a call — and a model built for a decision that no longer exists keeps producing correct predictions for it.
A retention team still makes a weekly call list from the model's ranking, with an offer available, at the capacity the threshold was set for.
holds when The decision owner named in the formulation is still in the role, the offer budget is unchanged, and the list is being consumed — the action log shows calls made against ranked users each week.
breaks when The team is reorganised, the offer is withdrawn, or an automated in-app prompt replaces the call — the action moves to a different moment with a different cost, and the top-k at weekly capacity is no longer the decision.
respond Reopen the formulation from the decision. A new action usually means a new moment, a new cost structure, and often a new horizon — a new target rather than a retrain.
How to build it
Most important first.
- Start with the decision and its owner in the room, and write the six answers down as a one-page document before opening a notebook (Decision Before Model).
- Fix the prediction moment as a timestamp rule — "the first of each month, for subscribers active on that day" — because it decides which features are legal and which split is honest (Target Definition, Time-Based Split).
- Derive the target from the decision, not from the columns that happen to exist: the retention team acts within a month, so the horizon is thirty days (Label Construction).
- Price the two mistakes with the decision owner before any metric is chosen, and choose the metric and threshold from the prices (Prediction vs Decision, Threshold Selection).
- Check that the label is observable for the whole population within the horizon, and that the action taken will not corrupt it (Label Leakage, Feedback Loops).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The number the decision owner cares about: subscribers retained per retention call, against the two-week rule as the baseline. If the model does not beat the rule on this number, it does not matter what it scores on any other.
- Precision at the retention team's weekly call capacity — the top-k they can actually contact — on a time-based split with the prediction moment respected. This is the offline number that maps to the decision.
- Do not measure AUC over all subscribers. It rewards ranking users the team will never call, and it treats the two mistakes as equal when the offer price says they are not.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The retention team still makes a weekly call list, still has an offer to make, and still acts on the model's ranking rather than the two-week rule.
- A prediction on the first of the month is still early enough to act on, and the thirty-day horizon still matches the window in which an offer can change the outcome.
- The cost of a wasted offer and the cost of a missed cancellation are still roughly what they were priced at when the threshold was chosen.
- Cancellations are still recorded as
cancelled_atevents within the horizon, for every subscriber, including those contacted.
- Offline: the one-page formulation is reviewed by the decision owner and the data owner before training; every feature is checked against the prediction moment; the evaluation uses the decision's top-k metric on a time-based split.
- Online: the fraction of the weekly list that the team actually contacts, and the retained-per-call number against a held-out slice served by the old rule.
- Over time: a quarterly re-read of the six answers with the decision owner; any changed answer reopens the formulation before anyone touches the model.
What can go wrong
- The six questions are answered, and the answer to "what action follows" is "we'll figure it out"; the model ships, the retention team has no offer budget, and the scores are never used.
- The decision owner changes: the retention team is folded into support, the weekly list becomes a daily in-app prompt, and the prediction moment, action and costs all change without the model changing.
- Formulation is done well and then reopened at every meeting; the target drifts from thirty days to sixty to "engagement", and no dataset survives long enough to train on.
- A week of formulation before any modelling is a week in which a notebook could have shown a promising number, and on the problems where the obvious formulation was right it is a week lost.
- Fixing the prediction moment early discards features that would have made an impressive offline number — the ones that leak — and the formulated model looks worse on paper than the naive one.
- Pricing mistakes with the decision owner forces a conversation about the offer budget that the growth team may not want to have yet.
- "We have the data, so we can predict churn." Having cancellation timestamps is not having a target; the target is the event, at a horizon, from a moment, for a decision. The same column supports a dozen targets and most of them are useless to the retention team.
- "Formulation is a product-management task, the model is ours." The formulation decides which features are legal, which split is honest and which metric matters. An engineer who does not own it inherits its mistakes as a model that cannot be fixed.
- "Start with a quick model to see if there is signal." The quick model finds signal in the leaked features and the trial users, and the number it produces becomes the expectation the formulated model then fails to meet.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe six questions apply to every predictive system; for a ranking or forecasting problem "event" becomes "quantity" or "order", and the moment, action and cost questions are unchanged.
- DOMAIN-SPECIFICSubscription churn has a clean event and a clear horizon; in medicine the event may be a diagnosis that is itself uncertain, and in fraud the action changes whether the label is ever observed — the questions are the same, the difficulty of answering them is not.
Where the depth lives
This domain teaches the model and hands the rest off by name.