Prediction vs Decision
P(churn) = 0.78 is a prediction. "Offer a retention discount?" is a decision. The model produces the first; a threshold, a cost and a policy turn it into the second, and none of those three lives in the model.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model outputs a probability. What has to happen between that number and the action the business takes, and why must those steps be designed and owned separately from the model?
The retention lead: "The model says this customer is 78% likely to churn. Do I send the discount or not? And why did last month's batch send discounts to people who were never going to leave?"
The model outputs a probability; if it is above one half the customer is "a churner", so send the discount. The model has decided; the business just executes.
One half is the default cut-off for a symmetric loss. The discount costs a month of revenue and a lost subscriber costs many months, so the decision-optimal cut-off is somewhere else entirely, and last month's batch spent the budget on the wrong region of the score distribution.
- One half is the default cut-off for a symmetric loss. The discount costs a month of revenue and a lost subscriber costs many months, so the decision-optimal cut-off is somewhere else entirely, and last month's batch spent the budget on the wrong region of the score distribution.
- The probabilities were not calibrated: the model's "0.78" was produced on a training set where positives were up-sampled, and in the true population that score corresponds to a much lower rate. The cut-off was applied to a number that did not mean what it said (Calibration).
- The threshold sends a discount to a subscriber who was going to stay and would have paid full price, and to one who was going to leave regardless of the discount. Both are false positives to the decision, and only one of them is a false positive to the model.
- When the offer budget was cut in half, the threshold stayed at one half, so the batch overran the budget and the last third of the list was silently dropped in arbitrary order.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The model predicts P(cancel within 30 days) for each subscriber at the monthly snapshot. That probability is the prediction, and it is what the model was trained and evaluated to produce.
- The decision is whether to spend a one-month discount on this subscriber. Its inputs are the probability, the cost of the offer, the value of a retained subscriber, the offer budget, and any rule the business imposes — none of which the model knows.
- Monthly snapshots of subscriber state with the thirty-day cancellation label; an offer log recording who was sent a discount, when, and whether they stayed; the subscriber's plan price and expected remaining lifetime.
- The offer log shows last month's batch went to everyone above a probability of one half, chosen because it was the default.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A classifier trained with a probabilistic loss produces a score that, if calibrated, estimates P(event | features). That estimate is the prediction and its quality is a property of the model. The decision is a function of the prediction *and* of quantities outside the model: the cost of acting, the value of the outcome, the budget, constraints such as "never discount twice in a quarter".
- The simplest decision rule is expected value: act when P × (value of preventing the event) exceeds the cost of acting. Solving that for P gives a threshold, and the threshold moves whenever the costs move, without any change to the model. A fixed capacity replaces the threshold with a top-k; a budget with a knapsack over expected value.
- A prediction is therefore never automatically the business action. The same model serves a generous budget and a tight one, a cheap offer and an expensive one, with different thresholds, and the confusion matrix at each threshold is a different business outcome from the same predictions.
Two mistakes with two prices
The confusion matrix is where the prediction meets the decision. Each cell is a business outcome with a price, and the prices are not symmetric: a wasted discount costs one month of one subscriber's revenue, while a missed cancellation costs that subscriber's remaining lifetime. A metric that adds the two cells together as if they were the same — accuracy, and F1 at a fixed ratio — is answering a question the business did not ask.
The counts below are at the default cut-off of one half. Moving the threshold moves subscribers between the cells, and the right threshold is the one that minimises the total price, not the one that maximises any single metric.
At this cut-off there are two wasted offers for every subscriber saved. Whether that is acceptable depends entirely on the ratio of the two prices, and the model cannot tell you.
From a probability to an action
Written out, the decision rule is short and contains no model. It contains the probability, the value of preventing a cancellation, the cost of the offer, and a budget. The threshold falls out of the first three and the budget turns it into a top-k when the list is longer than the money.
Everything in this function is business configuration and changes on a business timescale. Keeping it out of the model means a price change is a config change, not a retrain, and a wrong batch can be traced to the line that caused it.
1def decide(p_churn, plan_value_remaining, offer_cost, uplift, budget, candidates):2 # p_churn: calibrated P(cancel within 30 days), from the model3 # uplift: fraction of would-be cancellers the offer actually retains4 # Expected gain of offering = p * uplift * value_remaining - offer_cost5 scored = []6 for user in candidates:7 gain = p_churn[user] * uplift * plan_value_remaining[user] - offer_cost8 if gain > 0:9 scored.append((gain, user))10 # The break-even threshold, for a fixed value: p* = offer_cost / (uplift * value)11 # A budget turns the threshold into a top-k by expected gain.12 scored.sort(reverse=True)13 max_offers = budget // offer_cost14 return [user for _, user in scored[:max_offers]]Two things the model does not know appear here: uplift — the offer only matters for subscribers it would actually change — and budget. Both move the cut-off, and neither is a modelling question.
The threshold is an assumption too
The policy layer removes the threshold from the model but not from the system. A threshold derived from prices is correct only while the prices hold and the score means what it meant. Both drift: the offer is repriced, the budget is cut, and the next retrained model produces scores on a different scale for the same subscribers.
So the threshold needs the same treatment as any other assumption: written down with what it was derived from, and monitored for the conditions under which the derivation stops holding.
The cut-off in the policy layer is the one that minimises expected cost for the current offer price, retained value, uplift and budget, applied to scores that are calibrated for the current population.
holds when The threshold is re-derived on every price, budget or model change, and production calibration — score against realised thirty-day cancellation rate — is checked as labels arrive.
breaks when The offer is repriced without a policy update; the model is retrained and its score distribution shifts; the population changes so that a given score no longer corresponds to the same cancellation rate.
respond Re-derive the threshold from current prices and, if calibration has moved, recalibrate the scores before touching the model; only if ranking quality itself has decayed is a retrain the answer.
How to build it
Most important first.
- Separate the prediction service from the decision policy in code and in ownership: the model produces P, a policy layer owned with the business turns P into an action using costs, budget and rules (Thresholding).
- Derive the threshold from the costs, and re-derive it whenever the costs, the budget or the offer change; treat the threshold as configuration, versioned and logged with each decision (Threshold Selection).
- Calibrate the probability before using it in an expected-value rule, and check calibration in production as labels arrive; an uncalibrated score can still rank but cannot price (Calibration).
- Log the prediction, the threshold, the policy version and the action separately, so an incident can say whether the model or the policy was wrong (Prediction Logging).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The decision's outcome: net revenue retained per discount sent, against a control group that received no offer. This number depends on the threshold and the policy as much as on the model, and it is the one the retention lead is paid on.
- For the model alone: ranking quality and calibration on a time-based split. These say whether the prediction is good; they say nothing about whether the threshold is.
- Do not report "accuracy at one half" as the model's quality. It is the outcome of one arbitrary decision rule applied to the predictions, and a different rule would give a different number from the same model.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The model's probabilities are still calibrated in the population being scored — a score of a given value still corresponds to roughly that rate of cancellation.
- The costs the threshold was derived from — offer price, retained value, budget — are still current, and the threshold has been re-derived since they last changed.
- The policy layer is the only path from prediction to action; nobody is reading the raw scores and applying their own cut-off.
- The offer still changes outcomes for the subscribers it targets; the decision is not spending on those who would leave regardless.
- Offline: for the chosen threshold, report the confusion matrix and the expected cost using the business's prices; sweep the threshold to show the cost curve, and check that the chosen point is near its minimum.
- Online: the realised offer count against budget each batch; calibration of scores against thirty-day outcomes as they arrive; net revenue per offer against the control slice.
- Over time: re-derive the threshold on every change to costs, budget or model version, and alert when the selected fraction of subscribers moves without a policy change.
What can go wrong
- The policy layer exists and the threshold is set once from the costs at launch; the offer price changes and the threshold does not, and the batch quietly becomes unprofitable.
- The model is retrained, its score distribution shifts, and the old threshold now selects a different fraction of subscribers; the policy was correct for the previous model's scores (Prediction Drift).
- A subscriber who would have stayed at full price is discounted because the policy has no notion of "would this action change the outcome" — the model predicts churn, not the effect of the offer (Causality vs Prediction).
- A separate policy layer is a second component to version, test and explain, and the business now has to state its costs explicitly rather than leaving them implicit in "seems high".
- Calibration adds a fitted step whose own assumptions can drift; a ranking-only use of the model would not need it.
- A control group that receives no offer is a deliberate, measurable loss paid so the decision's value can be measured at all.
- "The model says 78%, so the customer is a churner." The model says that in the training population subscribers with these features cancelled at roughly that rate. Whether to act on it depends on what acting costs and what it gains, which the model does not know.
- "Tune the threshold to maximise F1." F1 treats a wasted discount and a lost subscriber as the same mistake at a fixed ratio. The business has prices for both; use them.
- "The model decided to send the discount." The model produced a number; a threshold someone chose turned it into an action. When the batch is wrong, both need to be examined, and usually the threshold is the one nobody owns.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe separation between a calibrated prediction and a cost-derived action holds for any classifier feeding a decision; for ranking tasks the "threshold" is a cut-off in the list, and for regression it is a tolerance, but the ownership argument is the same.
- SIMPLIFIEDThe confusion matrix counts and the score values in this lesson are invented to show the shape of the argument; the cost arithmetic is illustrative and no real subscriber base or offer is being measured.
- CONTESTEDA serious position holds that calibrating and then thresholding is over-engineered: choose the threshold directly by sweeping it against the business cost on validation data, and skip calibration entirely, since the sweep already absorbs any miscalibration. That works whenever the costs are fixed and the score distribution is stable; the argument for calibration is that when costs or budgets change, a calibrated score lets the threshold be re-derived on paper instead of re-swept on data.
Where the depth lives
This domain teaches the model and hands the rest off by name.