MSE, RMSE and MAE
Squared error punishes a large miss quadratically and answers in squared units; RMSE restores the units but keeps the outlier sensitivity; MAE is the median-like metric that treats every unit of error the same.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Two delivery-time models have nearly the same RMSE and very different MAE. Which one is better, and what does the difference in the two numbers tell you about how they miss?
A logistics team wants to show customers an estimated delivery time. They have two candidate models. One is consistently a few minutes off; the other is usually closer but occasionally predicts forty minutes for a delivery that takes three hours. Support tickets come from the second kind of miss, and the team cannot decide which model to ship.
Compute the mean squared error on the validation set, because that is what the loss function was, and pick the model with the smaller number. The metric and the loss agree, so the model that minimised one minimises the other.
The model with the better MSE got there by hedging toward the tail: it predicts a little longer for everyone so that the rare three-hour order costs it less. Customers see systematically pessimistic estimates, the product team sees conversion fall, and MSE never showed that trade because it was made to reduce MSE.
- The model with the better MSE got there by hedging toward the tail: it predicts a little longer for everyone so that the rare three-hour order costs it less. Customers see systematically pessimistic estimates, the product team sees conversion fall, and MSE never showed that trade because it was made to reduce MSE.
- MSE is in squared minutes. Nobody on the team can say whether an MSE of two thousand is good, so the number is reported and not interpreted; the decision gets made on a gut reading of a few examples.
- A handful of label errors — a courier tapping "delivered" the next morning — dominate the metric. The model comparison is being decided by the rows least likely to be real.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the number of minutes between order confirmation and hand-off to the customer. The label is the observed elapsed time from the courier app, which is missing or wrong when the courier forgets to tap "delivered".
- The prediction is shown to the customer as a single number, so the decision downstream is not a threshold but an expectation the customer will hold the company to.
- One example is one completed order: restaurant, distance, hour, day, courier load at dispatch, and weather at dispatch. Around a million rows over a year.
- The label distribution is right-skewed. Most orders arrive in twenty to fifty minutes; a small tail takes hours because of a restaurant outage or a courier reassignment, and those tail rows carry a disproportionate share of the squared error.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Mean squared error averages the square of each residual. Squaring means a residual of 60 minutes counts as much as 36 residuals of 10 minutes, so the metric is dominated by whichever examples have the largest errors, and a model minimises it by moving its predictions toward the conditional mean of the target — which, on a skewed distribution, is above the typical value.
- RMSE is the square root of MSE. It is in the target's units, which makes it readable, but the square root is applied after the averaging, so the ranking of models by RMSE is identical to the ranking by MSE and so is the outlier sensitivity. RMSE is MSE with better labelling.
- Mean absolute error averages the absolute residual. Every unit of error costs the same regardless of how large the miss already is, and the value that minimises it is the conditional median. On a skewed target, the MAE-optimal prediction sits where most deliveries actually land, and the tail is accepted as a tail.
What squaring does to a residual
A residual is a prediction minus its label. MSE squares each one before averaging, so the error from a single order that ran three hours late contributes as much as the error from dozens of orders that were ten minutes off. The metric is not describing the typical order; it is describing the worst ones, with the typical ones as background.
The consequence for a model trained on it is a pull toward the mean. On a right-skewed target the mean sits above the median, so an MSE-trained model over-estimates the typical order in order to be less wrong about the tail. That is a rational response to the loss. It is not what the customer wants to see.
1import numpy as np2 3def metrics(y, yhat):4 r = yhat - y5 mse = np.mean(r ** 2) # squared units; dominated by |r| large6 rmse = np.sqrt(mse) # target units; same ranking as mse7 mae = np.mean(np.abs(r)) # target units; every unit of error equal8 return mse, rmse, mae9 10# residuals with one heavy miss: nine orders 5 min off, one 180 min off11y = np.array([30] * 10)12yhat = np.array([35] * 9 + [210])13print(metrics(y, yhat)) # mse and rmse are set almost entirely by the last row14 15# the tail row removed: the same model on "normal" orders16print(metrics(y[:9], yhat[:9]))Run it with and without the last row. MAE moves a little; RMSE moves by a multiple. The ratio between them is the cheapest tail-weight measurement you will ever compute.
Same RMSE, different products
Two models with equal RMSE can have very different MAE, and the difference is not a rounding detail. A low MAE with the same RMSE means the model is close on most orders and pays for it with a few large misses; a higher MAE means the errors are spread evenly. The first model produces confident estimates and occasional angry customers; the second produces slightly vague estimates and few surprises.
Neither metric can tell you which of those the business prefers. What they can do, together, is show that the choice exists — which the single number the team started with had hidden.
| Option | Quality | Interpretability | Operational | Note |
|---|---|---|---|---|
| MSE | Tail-sensitive, squared units; the natural training loss, the worst reporting metric. | |||
| RMSE | Identical ranking to MSE with readable units; still decided by the largest residuals. | |||
| MAE | Every unit of error equal, median-seeking, robust to a few bad labels; blind to how bad the worst miss was. |
caveat The quality column is deliberately flat because the metrics do not differ in quality — they differ in which errors they count. A high score would imply one is the correct metric, and that depends entirely on the cost of a miss, which the matrix cannot see.
The metric assumes a tail shape
Choosing MAE because the tail was light, or RMSE because the tail mattered, encodes an assumption about the residual distribution. The distribution is a property of the data in production, not of the model, and it moves: a new city, a restaurant partner with unreliable kitchens, a courier-app change that starts recording next-morning hand-offs as elapsed time.
So the metric choice has a monitor attached. If the ratio between RMSE and MAE on the delayed production labels drifts, the tail has changed, and the metric that was appropriate at launch may no longer be measuring what the business feels.
The distribution of production residuals has roughly the tail weight it had on the validation set, so the metric chosen at launch still weights errors the way the business does.
holds when The mix of cities, restaurants and hours stays stable, and the label pipeline records elapsed time the same way it did when the validation labels were built.
breaks when A new region with long routes joins; a partner's kitchen becomes unreliable; the courier app changes how "delivered" is recorded and next-day taps enter the labels as multi-hour deliveries.
respond Look at the tail rows before touching the model. If they are label errors, fix the label pipeline; if they are real, decide again whether the business wants the tail counted, and then consider retraining.
How to build it
Most important first.
- Start from the cost of a miss (Choosing a Regression Metric). If a large miss is catastrophically worse than several small ones — a stock-out, a capacity breach — squared error is the honest metric. If a miss costs roughly per unit, MAE is.
- Report both, and report the gap. RMSE well above MAE is a direct measurement of how heavy the error tail is; two models with the same RMSE and different MAE miss in different ways, and that difference is the decision.
- Look at the residual distribution, not the summary. A histogram of residuals on the validation set, split by the tail rows, shows what the number averages away (Residuals & Assumptions).
- Clean or cap the labels before comparing models on a squared metric, or the comparison is a comparison of how each model handles a dozen bad rows (Label Quality).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- MAE at the level of the customer-facing number, because a minute of error costs roughly a minute of customer patience. This is the metric that maps to the support-ticket rate.
- RMSE alongside it as a tail detector — a rising RMSE with a flat MAE means the rare catastrophic miss is getting worse while the typical order is fine.
- Do not decide on MSE alone. It has no units a stakeholder can read, and its ranking is the same as RMSE's anyway.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The residual distribution in production has the same tail weight as in validation. A metric chosen for a light tail is the wrong metric once the tail thickens.
- The cost of a miss is roughly symmetric and roughly per-unit — the premise under which MAE maps to business cost. If under-promising and over-promising cost differently, neither MSE nor MAE is the right metric.
- The labels that reach the metric are true elapsed times. A change in the courier app that alters how "delivered" is recorded changes the metric without changing the model.
- Offline: compute MAE and RMSE on the validation set with and without the top one percent of residuals. If the model ranking flips, the decision is being made by the tail, and the tail needs its own examination.
- Online: log prediction and observed elapsed time per order and compute both metrics weekly on the delayed labels, per city.
- Over time: track the RMSE-to-MAE ratio. A ratio that climbs is the tail thickening, which is a different problem from the typical error growing.
What can go wrong
- The metric is computed on labels that include the courier-app failures, so both models are being scored against a target that is partly noise, and the squared metric amplifies exactly those rows.
- The team switches to MAE, the model learns the median, and the estimates become honest for most orders and too optimistic for the tail — the customers who wait three hours were told forty minutes. The tail did not go away; it stopped being measured.
- Production traffic shifts toward a new city with longer distances. Both metrics rise together and nobody can tell whether the model degraded or the target distribution moved (Data Drift).
- Optimising for MAE gives up on the tail by construction. The model will be right about most orders and confidently wrong about the ones that generate complaints.
- Optimising for MSE buys tail sensitivity by biasing every prediction toward the mean, which on a skewed target is a systematic over-estimate for the typical order.
- Reporting two metrics doubles the chance of a ranking disagreement, which is the point, but also means someone has to own the decision when they disagree.
- "RMSE is more accurate than MAE because it uses more information." It uses the same information with a different weighting. It is not more accurate; it is more sensitive to large residuals, which is a choice, not a virtue.
- "The two models have almost the same RMSE, so they are interchangeable." Same RMSE with different MAE means one is consistently a little off and the other is usually right and occasionally very wrong. They are different products.
- "MSE is what we trained on, so it is the right thing to evaluate on." The training loss is an optimisation convenience. The evaluation metric is supposed to be the cost of being wrong, and they only coincide when squared error is that cost.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat squared error targets the conditional mean and absolute error the conditional median follows from the definitions, whatever the model family or the domain.
- DATA-SPECIFICOn a symmetric, light-tailed target the three metrics rank models almost identically and the choice barely matters; the divergence appears with skew and heavy tails, which is most real-world durations, revenues and counts.
- SIMPLIFIEDThe numbers in this lesson — minutes of error, the scale of an MSE — are for the shape of the argument, not measurements of any real delivery system.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Statistics — that squared loss estimates the conditional mean and absolute loss the conditional median is a fact about loss functions in general, and the extension to quantile loss for asymmetric costs is where the statistics literature picks up.