Baselines Are Mandatory
A metric with nothing to compare it to is a number, not a result. Before anything complex: a rule, a constant predictor, a linear model, a shallow tree — and the question of whether the proposed model beats a useful one by enough to pay for itself.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model reports a strong validation metric. Compared to what — and what is the simplest thing that would have scored nearly as well?
A logistics company wants to predict which deliveries will be late so dispatch can warn customers. A contractor delivered a gradient-boosted model with a validation report and a recommendation to deploy. Nobody can say what the dispatchers' current rule of thumb — "anything routed through the north depot after 3pm" — would have scored on the same data.
The model has a validation report. The metric is good. Deploy it. A baseline is a formality that would only show a lower number, and everyone already knows the model will beat a coin flip.
The dispatchers' rule, scored on the same holdout, recovers most of the late deliveries the model finds. The model's margin over it is inside the noise of one month of data, and it costs a feature pipeline and a serving service to get it.
- The dispatchers' rule, scored on the same holdout, recovers most of the late deliveries the model finds. The model's margin over it is inside the noise of one month of data, and it costs a feature pipeline and a serving service to get it.
- The class is imbalanced, so the constant "never late" predictor scores high on accuracy, which is what the report led with. The report compared the model to nothing and so could not show that the metric it chose was uninformative (Accuracy Under Imbalance).
- Six months later the model is retrained and the metric drops. Nobody can say whether the model got worse or the month got harder, because there is no baseline scored on the same new month to anchor the comparison.
- The team is asked to justify the serving cost. The only honest answer is "we do not know what we gained", because nothing simpler was ever measured.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a delivery will arrive after its promised window. The label is the recorded arrival time against the window, observed at delivery.
- The decision is whether to send a proactive delay notification, which costs some goodwill when wrong and saves a support call when right.
- One example is one delivery at dispatch time: route, depot, time of day, driver load, weather, package count.
- Late deliveries are a minority of the volume, concentrated in a few depots and a few hours, which is exactly what the dispatchers' rule already captures.
- Six months of history with a time-based holdout on the last month.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A baseline is a predictor whose performance is a reference point: cheap to build, cheap to understand, and scored on exactly the same split with exactly the same metric as the candidate. Its number turns the candidate's number into a difference, and a difference is what decides.
- Baselines ascend in cost and capacity: the constant predictor (majority class, mean) shows what the metric rewards for free; the existing rule shows what the business already achieves; a linear model shows what the features contain without interactions; a shallow tree shows what a few interactions add. Each step that fails to improve is information about the problem.
- The margin over the baseline, not the candidate's absolute number, is the quantity with a cost attached. A model earns its serving cost, its monitoring and its retraining pipeline only by the amount it beats the thing the business would otherwise do.
A ladder of simple things
The baselines climb in capacity. The constant predictor knows only the label distribution; it shows what the metric gives away for free. The rule knows what the business knows; it shows what must be beaten. The linear model knows the features; it shows what they contain on their own. The shallow tree knows a few interactions. The candidate sits at the top and is only interesting by its distance from the rung below.
Each rung that fails to improve on the one below it is a finding. If the linear model matches the tree, the interactions are not there. If the rule matches the linear model, the features add little beyond what dispatch already knows. That is knowledge about the problem, obtained cheaply, before the expensive model is built.
- 1Constant predictor
Majority class or mean on the holdout, scored on the chosen metric.
fails by Scored on accuracy, which it wins; the finding that accuracy is the wrong metric is missed because the row is read as "trivially low".
- 2Existing rule
The business's current heuristic as code, on the same holdout.
fails by Transcribed from memory rather than from what dispatch actually does; the baseline is weaker than reality.
- 3Linear model
Regularised logistic or linear regression on scaled features, same encoding as the candidate.
fails by Unregularised on raw columns with a default encoding; loses badly and proves nothing.
- 4Shallow tree
A depth-limited tree to see whether a few interactions add anything.
fails by Grown without a depth limit and overfit; now it is a bad complex model, not a baseline.
- 5Candidate
The proposed model, in the same table, same split, same metric, same threshold policy, with intervals.
fails by Reported alone with a strong number and no rows above it.
The margin is the result
A validation metric is a number about one dataset and one split. The same number is a result only next to what a simpler predictor scored on the same data. The candidate's value minus the strongest useful baseline's value, with an interval, is what the deployment decision can actually be made on — because that is the quantity that has a cost on the other side.
The cost includes everything the baseline does not need: the feature pipeline, the serving service, the GPU if there is one, the monitors, the retraining runs, and the engineer on call for all of it. A rule the dispatchers run in their heads costs none of that. The model must beat it by more than that bill.
Candidate's recall at the notification budget reported as strong on the last-month holdout; no other row in the table.
Notifications go out; the support-call reduction is real but small. When the dispatchers' rule is finally scored in shadow, it captures most of the same late deliveries. The model's incremental value is a handful of warnings a day, against a serving service and a pipeline.
- 1The late deliveries are concentrated in a few depot-hour combinations the rule already names, so most of the signal was in the rule and the model found little beyond it.
- 2The one month of holdout gives a wide interval on recall; the margin over the rule, once computed, is inside it.
- 3The model genuinely does find some late deliveries the rule misses — the margin is real, just not worth the cost — which is the outcome the baseline would have shown before deployment.
Baselines stay in the pipeline
The comparison is not a one-off gate. A retrain that scores lower than last month's may be a worse model or a harder month, and only a baseline re-scored on the same new holdout can tell which. If the rule also dropped, the month was harder. If the rule held, the model regressed.
So the baselines are rows in every evaluation report the pipeline produces, not a notebook from the original project. That is the difference between "the model is at X this month" and "the model is at X, the rule at Y, and the margin has been shrinking for three retrains" — the second is a signal, the first is a number.
The candidate continues to beat the strongest useful baseline by a margin worth its operating cost, on current data.
holds when Every retrain re-scores the baselines on the new holdout; the rule is kept current with what the business actually does; the cost side is re-checked when volume or latency requirements change.
breaks when The world moves toward what the rule captures; the model degrades while the rule does not; the serving cost grows; the business improves its rule and nobody updates the baseline.
respond If the margin is inside the interval for several retrains, the honest move is to consider serving the baseline; if the model regressed while the rule held, debug the model, not the month.
| Baseline | What it knows | What beating it shows | What losing to it shows |
|---|---|---|---|
| Constant (majority / mean) | The label distribution | The metric rewards something beyond prevalence | The metric is wrong for this class balance |
| Existing rule | What the business already does | The model adds to current practice | The project is not needed |
| Linear model | The features, additively | Interactions or non-linearity carry signal | The candidate's capacity is unused; ship the linear model |
| Shallow tree | A few interactions | Deep interactions or many features matter | A small, legible model is enough |
How to build it
Most important first.
- Score the constant predictor first, on the chosen metric. If accuracy is high for "never late", accuracy is not the metric (Majority Class and Mean Predictor).
- Score the rule the business already uses, as code, on the holdout (The Rule Baseline). That is the number the model must beat by a margin worth the cost.
- Fit a regularised linear model on the same features (The Linear Baseline). If it is close to the candidate, the interactions the candidate learned are not worth much on this data.
- Put all of them in the evaluation report as rows in one table, same split, same metric, same threshold policy, with intervals (Beating the Baseline). Keep the baselines in the retraining pipeline so every future retrain is anchored.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The margin between the candidate and the strongest useful baseline, on the decision metric at the operating threshold, with an interval. That is the number that maps to "deploy".
- The cost side of the margin: serving latency and infrastructure per prediction, and the engineering time the pipeline needs (Inference Cost).
- Do not measure the candidate's absolute metric and call it a result. Without the baseline row it is uninterpretable.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The baseline's performance is stable enough that the margin measured at deployment time still describes the margin in production; if the rule the business used starts performing better or worse, the model's justification changes with it.
- The candidate and the baselines are scored on the same holdout, the same metric and the same threshold policy in every retrain, so the margin is comparable across time.
- The cost side of the margin — serving and maintenance — has not grown past the gain since the comparison was made.
- Offline: one table, rows for constant, rule, linear, shallow tree and candidate; columns for the decision metric at the operating threshold with intervals; every row on the same holdout (Metric Uncertainty).
- Online: run the rule in shadow next to the model and log both decisions; the difference in outcomes is the live margin (Shadow Deployment).
- Over time: the retraining pipeline re-scores every baseline on every new holdout; a shrinking margin is a signal to look at the model, a growing one at the baseline.
What can go wrong
- The baseline is built badly — a rule transcribed wrong, a linear model on unscaled features without regularisation — and loses by a margin that flatters the candidate. A weak baseline is worse than none because it looks like evidence.
- The baseline is scored on a different split, or on a different month, and the comparison is between two numbers that do not share a denominator.
- The baselines are built once for the deployment decision and dropped from the pipeline; two retrains later the comparison no one is making has silently reversed.
- Building good baselines is real work — the rule has to be transcribed exactly, the linear model has to be tuned honestly — and it is work that produces no model.
- A strong baseline sometimes ends the project, which is the right outcome and an unwelcome one for whoever proposed the model.
- Keeping baselines in the pipeline is one more thing to run and maintain at every retrain.
- "Everyone knows the model beats a coin flip, so the baseline is a formality." The coin flip is not the baseline. The rule the dispatchers already use is, and it is not obvious the model beats it by enough to matter.
- "The baseline scored lower, so the model is justified." Lower by how much, with what interval, at what cost per prediction? A margin inside the noise that costs a serving cluster is a loss.
- "We tried a linear model and it was worse, so we moved on." Was it regularised, on scaled features, with the same encoding as the candidate? A badly built baseline loses to everything and proves nothing.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a metric without a reference is uninterpretable holds for every task and model family; what changes across tasks is which constant predictor and which metric make the reference meaningful.
- SCALE-SPECIFICAt small volume the operating cost of a complex model is a rounding error and a small margin over the baseline is fine to ship; at high volume or tight latency the same margin can be worth less than the serving cost, and the baseline is what shows that.
- CONTESTEDA serious position holds that for problems where the modern approach is known to dominate — image classification with a pretrained backbone, large-scale ranking — building and tuning a linear baseline is ritual, and the time is better spent on the real model. That is fair for the linear step on those problems; the constant predictor and the existing rule still take an hour and still catch the wrong metric and the unnecessary project, and those are the ones this lesson insists on.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a baseline in the retraining pipeline is a regression oracle: the thing whose number is expected not to move, so that movement in the candidate can be attributed. Designing oracles that stay meaningful is a testing question this domain assumes rather than answers.