Majority Class and Mean Predictor
The constant predictor is the floor of every metric. Under imbalance it wins accuracy without looking at a single feature, and for regression it defines R² = 0 — which is why scoring it first is how you find out whether the metric means anything.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a predictor that ignores every feature score on our metric — and if that number looks good, what does it say about the metric?
A payments team is told their new fraud model "is 99.8% accurate". A second team forecasting warehouse demand reports a low RMSE. Neither report says what the number would be for a model that always predicts "not fraud", or always predicts last year's average. The finance reviewer wants to know whether either model learned anything.
Report the model's metric on the holdout. Accuracy is the metric everyone understands, RMSE is the standard for regression. A high accuracy or a low RMSE is a good model; a constant predictor is too trivial to bother with.
The always-negative predictor scores about the same accuracy as the fraud model, because the positive rate is one in a thousand. The model's reported accuracy is the class balance restated. It says nothing about whether any fraud was caught.
- The always-negative predictor scores about the same accuracy as the fraud model, because the positive rate is one in a thousand. The model's reported accuracy is the class balance restated. It says nothing about whether any fraud was caught.
- The mean predictor on demand achieves an RMSE that is the label's own standard deviation; the model's RMSE is only slightly below it. Relative to the mean, the model explained little of the variance — and the RMSE alone could not show that.
- Deployed on accuracy, the fraud model's threshold was set where accuracy peaks, which is close to "flag nothing". The review queue is empty and the chargebacks arrive on schedule (Threshold Selection).
- The demand model is used to plan stock; on the products with a seasonal swing its error is worse than last year's same-day value would have been, which nobody scored either.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Fraud: predict whether a transaction is fraudulent. The positive rate is roughly one in a thousand, and the label is a chargeback or an analyst confirmation, arriving weeks later.
- Demand: predict daily units per product. The label is realised sales.
- Fraud: one example is one transaction with cardholder aggregates and merchant features; positives are rare and the holdout is a later month.
- Demand: one example is one product-day with lags and calendar features; most products sell a fairly stable daily quantity with a seasonal swing.
- Both teams have a holdout; neither has scored a constant on it.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The majority-class predictor outputs the most common label for every input. Its accuracy is the majority fraction — for a positive rate of one in a thousand, it is wrong one time in a thousand. Any metric on which it scores well is a metric that rewards prevalence rather than discrimination, and accuracy is the canonical case (Accuracy Under Imbalance).
- The mean predictor outputs the training-set mean for every input. Its mean squared error on the holdout is (approximately) the variance of the label, and R² is defined as one minus the model's squared error over that quantity. So R² is zero for the mean predictor *by construction*; it is the comparison to the constant built into a metric (R² (Coefficient of Determination)).
- Both are floors. A model below the majority predictor on a discrimination metric, or with negative R², is worse than ignoring the features. A model barely above them has learned little, whatever its absolute number looks like.
The majority class wins accuracy for free
If one transaction in a thousand is fraud, a predictor that says "not fraud" to everything is right 999 times in a thousand. Its accuracy is 99.9%. A model that reports similar accuracy has told you the class balance and nothing about whether it caught any fraud. The confusion matrix shows it immediately: the constant's matrix has both positives in the false-negative cell and nothing in the true-positive cell.
The metric to use is one on which the constant scores badly and the difference between models is visible: recall and precision at the operating threshold, or the precision-recall curve. On those, the constant is at zero recall, and every real detection is a real difference.
Accuracy for this matrix is 99.9% and recall is zero. Any report that leads with accuracy has led with the number this predictor wins.
The mean predictor defines zero
For regression the constant is the mean of the training labels. Its squared error on the holdout is the variance of the label around that mean. R² is one minus the model's squared error divided by that quantity — so R² is exactly the model's improvement over the mean predictor, and the mean predictor scores zero by construction. A model with R² near zero has matched the constant; below zero, it is worse than ignoring the features.
The same idea generalises: any error metric becomes a skill score when divided by the constant's error. For demand, the more honest constant is often not the mean but the seasonal naive — the same day last week or last year — which knows the seasonality the mean does not, and is usually much harder to beat.
1import numpy as np2 3def majority_accuracy(y):4 p = y.mean() # positive rate5 return max(p, 1 - p) # always predict the majority class6 7def r_squared(y_true, y_pred):8 ss_res = np.sum((y_true - y_pred) ** 2)9 ss_tot = np.sum((y_true - y_true.mean()) ** 2) # the mean predictor's SS10 return 1 - ss_res / ss_tot # mean predictor -> 0; worse than mean -> negative11 12def skill(err_model, err_baseline):13 return 1 - err_model / err_baseline # 0 = no better than the baseline14 15# demand: score the model against BOTH constants16mean_pred = np.full_like(y_val, y_train.mean())17seasonal_naive = y_val_last_year_same_day18print(skill(mae(y_val, model_pred), mae(y_val, mean_pred)))19print(skill(mae(y_val, model_pred), mae(y_val, seasonal_naive)))Two skill numbers, not one. A model can show real skill over the mean and none over the seasonal naive — which means it learned the seasonality and nothing else, and last year's calendar would have done.
The floor moves
The constant's score is a property of the label distribution, so when the distribution changes the floor changes with it. If the fraud rate doubles, precision at a fixed threshold rises with no change in the model; if a product's demand variance grows, every error metric grows. A report that tracks the model's absolute number over time is tracking the floor as much as the model.
Keeping the constant in the pipeline turns this into a signal: a change in the floor is a change in the world, and the model's skill relative to the floor is the number that says whether the model is holding.
The class balance and label variance in production are close enough to the holdout's that the metric's floor — and therefore the meaning of the model's number — has not moved.
holds when Prevalence and label spread are monitored and stable; the operating threshold was set on a prevalence-robust metric; the constant is re-scored on every new holdout.
breaks when A fraud campaign doubles the positive rate; a product line changes its demand pattern; the holdout was a quiet month and production is not.
respond Re-read the model's metric as skill relative to the new floor before concluding anything about the model; then decide whether the threshold or the model needs to change.
What does ignoring the features look like for this task, and which metric exposes it?
when Positive rate far below half; fraud, defects, rare disease.
cost Majority predictor wins accuracy; use recall and precision at the operating threshold or PR AUC, and carry a threshold policy.
when Classes near even.
cost Majority predictor is at chance on accuracy; accuracy is informative and the constant is a low bar, so the rule baseline carries the comparison.
when Little seasonality or trend.
cost Mean predictor sets R² = 0; report R² or a skill score against the mean.
when Demand, traffic, anything with a calendar.
cost Mean is too weak a floor; seasonal naive is the honest constant and is often hard to beat, which is the point.
How to build it
Most important first.
- Score the constant first, on every metric in the report. Where it scores well, replace the metric: precision and recall at the operating threshold, or PR AUC, for the rare-positive case (PR AUC); MAE or error relative to a seasonal naive for demand.
- For regression, report R² or a skill score against the mean and against the seasonal naive — last period's value, or the same day last year — so the model's error is always a ratio to a constant's.
- For classification under imbalance, report the confusion matrix at the operating threshold with the business cost of each cell; the constant's matrix is all in one column, which makes the point visually (The Confusion Matrix).
- Keep the constant in the pipeline; it re-establishes the floor on every new holdout, and a floor that moves — the fraud rate doubling — is itself a signal (Prediction Drift).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Fraud: recall at the queue budget, and precision there; the constant scores zero recall, so any recall is a real difference.
- Demand: the ratio of the model's error to the mean predictor's and to the seasonal naive's, per product group; a ratio near one is "learned nothing here".
- Do not measure accuracy under a one-in-a-thousand positive rate, or RMSE with no constant next to it. Both report the label distribution and call it the model.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The class balance or label variance in production is close to the holdout's, so the floor the constant established is still the floor; if the fraud rate doubles, every prevalence-sensitive metric moves without the model changing.
- The operating threshold was chosen on a metric the constant does not win, so the deployed decision is not "flag nothing" dressed as a model.
- The report's metrics are the ones the constant was scored on, and the constant's row is still in the report the pipeline produces.
- Offline: for every metric in the report, the constant's value next to the model's; any metric on which they are close is removed from the decision.
- Online: the prevalence of positives in production against the holdout's; a monitor on the floor, because a moved floor is a moved metric (Ground-Truth Delay).
- Over time: the constant and the seasonal naive re-scored on every retrain holdout, with the model's skill relative to them tracked rather than the absolute number.
What can go wrong
- The constant is scored once and the metric is switched to PR AUC; later the positive rate changes and PR AUC moves with it, and the change is attributed to the model (Class Imbalance).
- R² is reported on the training set, where it can only be flattering, or on a holdout from a different regime where the label variance differs, so the denominator is not the one the model was fitted against.
- The seasonal naive is a strong baseline for demand and is never scored; the model beats the mean handily and loses to last year's same day on half the products.
- Replacing accuracy with precision and recall at a threshold means the report now has a threshold policy to defend, and two numbers instead of one.
- Skill scores relative to a baseline are less intuitive to a business reader than a single absolute error; the honest number needs an explanation.
- Scoring the seasonal naive needs a year of history per product, which new products do not have; for them the mean is the only constant available and the comparison is weaker.
- "99% accuracy means the model is good." At a one-in-a-thousand positive rate, the always-negative predictor gets 99.9%. The model's accuracy is the class balance, and the number that says whether it caught anything is recall at the threshold.
- "The RMSE is low, so the forecast is accurate." Low relative to what? The mean predictor's RMSE is the label's standard deviation; if the model is close to it, R² is near zero and the features explained little.
- "R² is zero, so the model is broken." R² is zero for the mean predictor by definition; a model at zero has matched the mean, not failed catastrophically — it has learned nothing beyond it, which is a different and clearer finding.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the constant predictor is the floor of every metric holds for every task and model family; which constant — majority class, mean, median, last value — and which metric it exposes as uninformative depends on the task and the label distribution.
- DATA-SPECIFICOn balanced classes accuracy and the majority baseline are informative and the point of this lesson mostly vanishes; the more skewed the positive rate, the closer accuracy is to the class balance and the more the report needs precision, recall or PR AUC at the operating threshold instead.
- SIMPLIFIEDThe positive rate, accuracy figures and the R² construction here are for the shape of the argument, not measurements; the relationship "majority accuracy equals the majority fraction" is exact, and the R² identity holds for the mean of the evaluation set, approximately for the training mean.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the constant predictor is the null hypothesis of the evaluation, and the practice of always testing against a null before believing a result is a statistics discipline this domain applies rather than teaches.