When Not to Use ML
A rule works; labels cannot be observed; the decision cannot use a probability; a wrong prediction has unbounded cost; the data does not exist at prediction time. Any one of these is a reason to stop, and the formulation is where you find out.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which properties of a problem make a learned model the wrong tool even when a good model could be trained, and how do you recognise each one before building it?
The CTO: "We have a data science team now. I want a list of everything in the product that could be a model." The team has five candidates and a quarter, and needs to know which ones to refuse.
Every problem with data and an outcome is a modelling problem. A model will beat a rule, an unobservable label can be approximated, a probability can be thresholded, and the risk can be managed with monitoring.
The spending-limit model matches the rule on the cases the rule handles and is wrong on the edge cases, because the rule *is* the policy; a learned approximation of a known function adds error, latency and monitoring for nothing.
- The spending-limit model matches the rule on the cases the rule handles and is wrong on the edge cases, because the rule *is* the policy; a learned approximation of a known function adds error, latency and monitoring for nothing.
- The attrition model is trained on who left; once managers act on its scores, the people it flags are given raises and stay, and the next training set records them as loyal. The label was observable only while nobody acted on the prediction.
- The refund model outputs a probability that the refund is legitimate; the decision is binary, has a legal standard, and the customer is entitled to an explanation — "the model gave you 0.4" is not one. The decision cannot consume a probability.
- The API shut-off model is right most of the time; the one time it shuts off a hospital's integration during an outage the cost is not a number the threshold can be set against.
- The demand forecast for the launch has no history for the product; every feature the model would use does not exist at prediction time, and the forecast is a prior dressed as a prediction.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Five candidate problems, each with a plausible target: flag orders over a spending limit; predict which employees will leave; auto-approve refunds; decide whether to shut off a customer's API access for abuse; forecast demand for a product launching next month.
- The lesson's target is the decision to *not* build, and the specific property of each problem that makes it right.
- The order and refund data is complete and labelled; employee attrition is labelled but the actions taken on predictions would change who leaves; API abuse is decided by a human review that most flagged accounts never receive; the new product has no history at all.
- The spending-limit rule is three lines and has been correct for years.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A learned model earns its place when there is a pattern in data that is too complex or too changeable to write down, a label that can be observed independently of the action taken, a decision that can act on a probability or ranking, mistake costs that are bounded and can be priced, and features that exist at the prediction moment. Each of these is a precondition, and each maps to a stage of the formulation where it is checked.
- When the pattern is already written down — a policy, a regulation, a threshold someone chose — a model is an approximation of a function you already have, and its errors are pure cost. When labels are only observable if no action is taken, the model consumes its own training signal. When the decision needs a justification, a probability is not an answer. When the cost of one mistake is unbounded, no threshold is decision-optimal. When the data does not exist at the moment, the model has nothing to condition on.
- These are not modelling difficulties. A stronger model, more data or a better metric does not address any of them, which is why they have to be caught in formulation — after it, every stage of the pipeline works correctly on a problem that should not have been posed.
Five reasons to stop
Each row is one of the CTO's candidates, the precondition it fails, and what to do instead. Two of the five are worth building; the other three are worth building only after something else changes. The point of the table is that the reason to stop is a property of the problem, not of the model, so no modelling effort can remove it.
Notice the "instead" column is never "nothing". A rule, a control group, a human process or an honest prior is the alternative, and each is measurable in its own right.
Does this problem meet the preconditions for a learned model?
when Never — a rule works. The limit is the policy, and the rule implements it exactly.
cost Ship and measure the rule; revisit only if the policy becomes too complex to write down.
when Only with a control group or an exploration policy, because acting on the prediction erases the label.
cost A slice of at-risk employees deliberately not intervened with, which is a real cost and needs sign-off.
when As a prioritiser for human review, not as the decision, because the decision has a legal standard and needs an explanation.
cost A review queue and an escalation path; the model saves reviewer time rather than replacing the reviewer.
when As a flag to a human, never as the action, because one wrong shut-off has unbounded cost.
cost A human on call and a measured override rate; latency to shut-off is bounded by the human, not the model.
when Not as a model — the features do not exist at the prediction moment. Offer a comparable-launch prior and say what it is.
cost An honest range instead of a number; a model once a few weeks of sales exist.
The label that disappears when you act on it
The attrition case is the subtlest and the most common. The label is observable today because nobody is acting on predictions. The moment managers use the scores, the flagged employees receive attention and some stay, and the next training set records them as negatives. The model has not become wrong — the label has become a record of the model's own effect.
This is a feedback loop at the formulation stage, before any model exists. The fix has to be designed in — a control group, an exploration policy, or a label that is observed independently of the action — and if none is acceptable, that is a reason to stop.
Whether an employee leaves is recorded the same way regardless of whether the model flagged them, so the training labels reflect the outcome and not the intervention.
holds when A control slice is never shown to managers; interventions are logged so treated and untreated outcomes can be separated; or the target is redefined as something the action does not change.
breaks when Every flagged employee is intervened with and the intervention works; the control group is removed to "help everyone"; interventions are not logged so treated negatives are indistinguishable from true negatives.
respond Stop retraining on the contaminated labels; rebuild the training set from the control slice and the intervention log; if no control slice exists, the model cannot be honestly retrained and should be retired until one does.
What the tradeoff matrix cannot see
Comparing the alternatives on the usual axes makes the rule look weak and the model look strong, and that is exactly the comparison that leads to building the wrong thing. The axes measure properties of a component; the preconditions are properties of the problem, and a component that scores well on every axis is still wrong for a problem that fails one.
The caveat is the lesson: no cell in the matrix can express "the label disappears when you act on it" or "one mistake is unbounded". Those are read off the formulation, or not at all.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Hand-written rule | Exactly right where the policy is the function; wrong on the residual it cannot express; needs no data and almost no operation. | ||||||
| Learned model, automated | Finds patterns the rule cannot; needs labels, monitoring, retraining and a serving path; cannot explain itself to a customer. | ||||||
| Model as prioritiser, human decides | Keeps explanation and unbounded-cost decisions with a person; bounded by the human's throughput; needs an override metric. |
caveat The scores describe components on a problem that meets every precondition. They cannot express that a rule scores five on quality when it *is* the policy, that a model's quality is undefined when its label is action-dependent, or that no latency or cost advantage matters when one mistake has no price ceiling. Read the preconditions first; the matrix is for what is left.
How to build it
Most important first.
- Run the five checks as the last step of formulation, before any data is pulled, and treat any failure as a stop rather than a caveat (Problem Formulation).
- Where a rule exists, ship the rule, measure it, and consider a model only for the residual the rule demonstrably gets wrong (The Rule Baseline).
- Where the label is action-dependent, design a control group or an exploration policy before the first model, or the second model has no honest data (Feedback Loops, Exploration vs Exploitation).
- Where the decision needs explanation or has unbounded cost, put a human in the loop with the model as a prioritiser, and design the escalation path first (Human Oversight, Explainability).
- Where the data does not exist at the moment, say so, and offer the prior — a comparable product's launch curve — as what it is (Cold Start).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For each candidate: whether a rule already achieves the decision's target on the decision's metric; whether the label rate would move if the prediction were acted on; whether the decision owner can act on a probability; the worst-case cost of one mistake; the fraction of proposed features available at the moment.
- The number that matters is the improvement over the rule on the decision's metric, net of the model's cost — and for three of the five candidates it cannot be computed because a precondition fails.
- Do not measure candidates by "how good a model could be trained". A near-perfect attrition model still poisons its own labels.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The rule that replaced the model is still the policy, and the residual it gets wrong is still small enough not to justify a model.
- The control group or exploration policy that keeps the attrition label observable is still in place and has not been "optimised away" because it costs something.
- The human review that owns the unbounded-cost decision is still reviewing rather than approving, measured by the rate at which it disagrees with the model.
- Offline: for each candidate, a written answer to the five checks signed by the decision owner, and for a shipped rule the same evaluation a model would have had.
- Online: for the human-in-the-loop cases, the override rate; for the control-group case, the label rate in the control slice against the treated slice.
- Over time: re-run the five checks when the decision, the regulation or the data changes; a candidate that was refused for a missing precondition may pass a year later, and one that shipped may fail.
What can go wrong
- The checks are run and a candidate fails one "slightly" — the refund decision "mostly" does not need explanation — and the model ships with the exception handled by a rule that grows until it is the decision.
- A rule is shipped instead of a model and nobody measures it either; the problem the model would have had is invisible in the rule too.
- The human-in-the-loop design becomes a rubber stamp within months, and the model is de facto deciding the unbounded-cost case with a person's name on it.
- Refusing three of five candidates makes the data science team look under-used to a CTO who wanted a list, and the refused problems are often the ones with the most visible business pain.
- Shipping a rule forgoes whatever residual improvement a model would have found on the cases the rule gets wrong.
- A control group for the attrition case means deliberately not acting on some at-risk employees, which is a real cost and an ethics question.
- "A model will always beat a rule eventually." A model approximates a function from data; when the function is already known and written down, the approximation can only add error. The model wins where the function is unknown or changes faster than the rule can be edited.
- "We can't observe the label, so we'll use a proxy." A proxy target is a different target, and the model will optimise it faithfully; whether that helps the real decision has to be shown, not assumed.
- "The risk is manageable with a conservative threshold." A threshold is chosen by balancing priced mistakes. When one mistake has no price ceiling, there is no balance to strike and the threshold is a guess with a number on it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe five preconditions apply to any learned component in any decision; they are stated for a classifier but a forecast, a ranker or a fine-tuned language model fails them in the same ways.
- CONTESTEDA serious position holds that this list is too conservative: a model can be a useful advisor where a rule is the policy, proxies for unobservable labels are how most real systems work, and human-in-the-loop designs handle the unbounded-cost cases well enough in practice. Each of those is true of a system designed around the limitation from the start; the argument here is only that the limitation must be found in formulation, because after it no stage of the pipeline can reveal it.
Where the depth lives
This domain teaches the model and hands the rest off by name.