The Rule Baseline
The heuristic the business already uses is the strongest baseline most models face, and the honest reason a model has to win by a margin: the rule is free to run, already trusted, and already in production.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does the rule the business runs today score on our holdout — and if we cannot say, what is the model being compared to?
A telecom retention team calls customers they think are about to leave. Today they call anyone who has filed two support tickets in a month. A churn model is proposed to "prioritise the call list". The team lead asks a fair question: "we already know who to call — how many more leavers would your model find, and how many fewer wasted calls?"
The rule is one feature and a threshold; any model with fifty features will beat it. Build the model, report its metric, and assume the rule is far below it. Comparing to the rule is a formality the team lead is asking for politely.
The rule, scored properly on the holdout, catches a large share of the leavers the model catches. Tickets are the strongest feature in the model, and the rule is that feature with a threshold. The model's margin is the contribution of everything else, and it is modest.
- The rule, scored properly on the holdout, catches a large share of the leavers the model catches. Tickets are the strongest feature in the model, and the rule is that feature with a threshold. The model's margin is the contribution of everything else, and it is modest.
- The rule was reconstructed as "two tickets in a calendar month" but the team runs "two open tickets in a rolling 30 days". The baseline in the report is weaker than the one in production, and the model's margin is inflated by the transcription error.
- The rule is free: no pipeline, no serving, no monitoring, no retraining, and every dispatcher understands it. The model's modest margin has to pay for all of that before the business is better off.
- The model is deployed and the team keeps running the rule alongside it "to be safe". The call list is now the union, the budget is exceeded, and nobody can say what either contributed (Decision Before Model).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict which subscribers will cancel within 60 days, to rank a call list of fixed size each week. The label is the cancellation event.
- The decision is who gets a call, under a fixed budget of calls per week; what matters is how many eventual leavers are on the list and how many calls are wasted.
- One example is one subscriber at the start of a week with ticket counts, usage, tenure, plan and payment history.
- The rule — two tickets in 30 days — is applied by the team from a report; its exact definition (calendar month or rolling 30 days? open tickets or any?) lives in the report's SQL, which was written by someone who left.
- Historical call lists exist, so the rule's past behaviour can be reconstructed — but the calls themselves changed outcomes for the people called (Feedback Loops).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A business rule is a model: a hand-fitted decision function over one or two features with a threshold chosen by experience. It encodes domain knowledge that the learned model has to rediscover from data, and on a problem where one feature dominates, rediscovering it is most of the model's performance.
- Scoring the rule means implementing it exactly as run — the same window, the same null policy, the same timing — and applying it to the holdout as a predictor, then computing the same decision metric at the same budget as the candidate. It gets a row in the same table.
- The rule's true cost is near zero, so the margin the model must clear is not "better than the rule" but "better by more than the model costs to run", including the cost of a serving incident that the rule could never have.
The rule is a model someone already fitted
"Call anyone with two tickets in a month" is a decision function: one feature, one threshold, chosen by people who watched customers leave for years. It is a model with a hand-tuned parameter, and on a problem where tickets are the strongest feature, it is not far from the best one-feature model there is.
Scoring it is the same as scoring any predictor: implement it, apply it to the holdout, compute the decision metric at the budget. What people skip is the implementation, because they assume they know the rule. The rule that is run and the rule that is described differ in the window, the null policy and the timing, and the difference changes the score.
1-- the report's own query, not the description of it.2-- note: rolling 30 days from the list date, open tickets only, and3-- customers on a promotional plan are excluded (a quirk nobody mentioned)4SELECT c.customer_id5FROM customers c6JOIN tickets t7 ON t.customer_id = c.customer_id8 AND t.opened_at > :list_date - INTERVAL '30 days'9 AND t.opened_at <= :list_date10 AND t.status = 'open'11WHERE c.plan_type <> 'promo'12GROUP BY c.customer_id13HAVING COUNT(*) >= 2;The promo exclusion and the open-only filter are the rule as run. The described rule — "two tickets in a month" — would score differently and is the baseline nobody runs. Transcribe the query, then test it against a real week's list.
Why the margin has to be large
The rule has no serving path, no feature pipeline, no monitor, no retrain, no artifact and no on-call. Its cost is a report. The model has all of those, and each is a place to fail. So the model does not need to beat the rule; it needs to beat the rule by enough that the extra leavers found pay for the extra machinery and the incidents the machinery will have.
This is the operating-cost form of the comparison, and it is why the rule is the most important baseline: it is the one whose cost is closest to zero. A small margin over the constant predictor is meaningless; a small margin over the rule is a loss.
The rule at the same budget might land at a somewhat smaller true-positive count. The comparison is the difference in that cell and the false-positive cell at a fixed list size, in the team's units — not either model's AUC.
"The churn model reaches a strong recall at the weekly budget." No rule row. The team lead's question — how many more leavers, how many fewer wasted calls — is unanswered.
"At 500 calls per week, the rule lists a certain number of eventual leavers; the model lists somewhat more and wastes somewhat fewer calls. That margin, with its interval, is the case for the model; the pipeline and serving cost is on the other side of it."
The second answer is the decision. It puts the gain and the cost in the same units on the same page and lets the team lead decide whether the extra leavers are worth a serving system.
The rule outlives the comparison
A rule that was worth scoring is worth keeping. As a feature it gives the model the domain knowledge directly; as a fallback it is what the serving path returns when the model or its features are unavailable; as a shadow it is the live measurement of the margin. A model that replaces the rule and discards it has thrown away its own baseline and its own fallback.
The rule also changes. The team moves to three tickets, or adds a payment-failure condition, and the baseline in the pipeline must move with it or the reported margin is against a policy nobody runs.
looks like Ordinary cancellation labels for every customer in the holdout weeks.
why it leaks Customers the rule flagged were called, and some who would have left stayed because of the call. Their label says "stayed", so the rule's flagged set looks less accurate than it was — the intervention wrote itself into the answer.
fix A randomised experiment between the two policies for a quarter; failing that, state the bias and treat the rule's offline number as a lower bound.
The rule scored in the pipeline is the one the team actually applies this quarter, with the same definition, window and exclusions.
holds when The rule is transcribed from the system that runs it, tested against a real week's list, and its owner is named in the pipeline so changes are communicated.
breaks when The team changes the rule informally; the report's SQL is edited; the pipeline's copy is a description rather than a transcription.
respond Update the baseline, re-score the margin, and re-check that the model still earns its cost against the new rule.
How to build it
Most important first.
- Get the rule from the system that runs it, not from the person who describes it. The report's SQL, the dashboard filter, the spreadsheet formula — transcribe that.
- Score the rule as a predictor on the holdout at the same budget: a call list of the same size, and the count of eventual leavers on it, for both.
- Give the rule a second life as a feature and as a fallback: the model can use it, and the serving path can fall back to it when the model is unavailable (Serving Fallbacks).
- State the margin in business units — leavers found per week, wasted calls per week — against the rule, with intervals, and put the model's running cost next to it.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Leavers on the list per week at the fixed budget, model against rule, on the same holdout weeks, with intervals. The difference is the entire case for the model.
- Wasted calls per week at that budget — the precision side, in the team's units.
- Do not measure the model's AUC against the rule's AUC. The rule produces a set, not a ranking, and the business runs a budget, not a curve; compare at the budget.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The rule scored as the baseline is the rule actually run, with the same definition, window and data source, and the pipeline updates the baseline when the business changes the rule.
- The margin of the model over the rule, in leavers found and calls wasted per week, remains larger than the model's operating cost as volume and staffing change.
- The labels used to score both were not produced by the rule's own interventions to a degree that biases the comparison; where they were, the bias is stated.
- Offline: the rule as code, unit-tested against a week of the team's actual call list to confirm it reproduces it; then scored on the holdout at the budget, in the same table as the candidate.
- Online: a randomised split of weekly call budget between rule-selected and model-selected customers for a quarter, measuring retention in each arm (A/B Testing Models).
- Over time: the rule stays in the retrain pipeline as a row and in the serving path as a fallback, so the margin is re-measured and the fallback stays exercised.
What can go wrong
- The rule's historical call lists altered outcomes: people the rule flagged were called and some stayed. The holdout labels for rule-flagged customers are biased toward "stayed", which makes the rule look worse and the model, which was trained on the same biased labels, look like it learned to avoid them.
- The rule changes — the team moves to three tickets — and the baseline in the pipeline still says two; the margin the model reports is against a rule nobody runs.
- The model wins by a real margin in the first quarter, then the ticket pipeline changes and both the model and the rule lose the feature; only the rule's owner notices, because the rule's failure is visible on a report.
- Transcribing the rule exactly means reading someone else's SQL and reproducing its quirks, including the ones that are bugs; scoring the "intended" rule instead is cleaner and wrong.
- A strong rule baseline shrinks the model's apparent contribution, which is honest and demoralising, and sometimes ends the project.
- Running the rule as a shadow or an experiment arm costs a share of the call budget on the weaker policy for a quarter.
- "The rule is one feature; the model has fifty; of course it wins." The one feature may carry most of the signal, and fifty features cost a pipeline. Score it and see.
- "The model catches leavers the rule misses." And the rule catches some the model misses. At a fixed budget, the question is the count on the list, not the existence of a difference.
- "We will run both to be safe." Then the budget doubles and neither can be evaluated. Choose, or randomise the split and measure.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICWhere operational experience is deep and the signal is concentrated — retention, fraud triage, maintenance scheduling — the existing rule is often within reach of a first model; in domains where no human can state the rule — image and speech tasks, large-scale ranking — there is no rule baseline to score and the constant and linear rungs carry the comparison.
- TASK-SPECIFICFor a fixed-budget list the comparison is a count at the budget; for a regression target the "rule" is usually last period's value or a seasonal average, and the comparison is error on the same horizon; for ranking it is the existing sort order and the comparison is the ranking metric at the cut-off.
- CONTESTEDA serious position holds that scoring the historical rule is meaningless because its past decisions changed the labels: customers it flagged were called and some stayed, so its holdout performance understates it and the comparison is biased in the model's favour by construction. That is correct and the bias is real; the reply is that the same bias sits in the model's training labels, that a randomised experiment is the clean answer, and that a biased rule row is still far more informative than no rule row.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the check that the pipeline's copy of the rule reproduces the team's real list is a golden test against a live system, and keeping such a test honest when the system it mirrors is edited by people outside the team is a testing question this domain assumes rather than answers.