Choosing a Regression Metric
Derive the metric from the cost of being wrong: is a ten-unit miss the same on a hundred-unit item as on a ten-unit item, are large misses catastrophic or merely bad, and is the decision actually a threshold on the forecast — in which case it is classification in disguise.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The team has tried MSE, MAE, R² and MAPE and each ranks the candidate models differently. Which questions about the business decide the metric, and when is the right answer that this is not a regression problem?
An energy retailer forecasts next-day consumption per customer to buy power on the day-ahead market. Under-buying means purchasing the shortfall at the volatile intraday price; over-buying means selling the surplus back cheaply. The team has four metrics, four rankings, and a trading desk asking which model to use.
Pick the metric the library reports by default, or the one the previous project used, and compare the models on it. The metrics all measure error; a model that is good on one is presumably good on the others.
The models disagree because the metrics weight errors differently, and the disagreement is not noise — it is each metric expressing a different opinion about which errors matter. Choosing one by default chooses that opinion without stating it.
- The models disagree because the metrics weight errors differently, and the disagreement is not noise — it is each metric expressing a different opinion about which errors matter. Choosing one by default chooses that opinion without stating it.
- The trading cost is asymmetric and depends on the aggregate, not the per-customer error. A model with the best per-customer MAE can have the worst aggregate bias, because per-customer errors that cancel cost nothing and per-customer errors that all lean the same way cost the intraday premium.
- For the demand-response customers, a forecast that is a little under the threshold and an actual a little over it is a small regression error and a large business loss. The regression metric says the model was nearly right; the penalty says it was wrong.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict tomorrow's hourly consumption per customer. The label is the metered reading, which arrives a day later and is occasionally estimated when the meter fails to report.
- The forecast is summed across customers into a purchase, so the decision is a quantity, and the cost of the miss is set by the market: a shortfall costs the intraday premium, a surplus costs the spread between purchase and resale.
- One example is one customer-hour: recent consumption, calendar, tariff, and weather forecast for the day. Millions of rows; the per-customer series are noisy, the aggregate is smooth.
- A subset of customers are on a demand-response programme where exceeding a threshold triggers a penalty. For them, the question is not "how much" but "will they exceed", which is a different problem hiding in the same dataset.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A regression metric is a loss over residuals, and every loss encodes two decisions: how the cost grows with the size of a miss, and whether the cost depends on the size of the thing being predicted. Squared error grows quadratically and is scale-dependent; absolute error grows linearly and is scale-dependent; percentage error grows linearly and is scale-inverted.
- The cost of a miss is also a function, and the metric is right when the two functions agree. If a large miss is catastrophic — a capacity breach, a market penalty — the cost curve is convex and squared error, or a loss that grows faster still, is honest. If a miss costs roughly per unit, the cost curve is linear and absolute error is honest.
- When the cost changes sign with the direction of the error, or jumps at a threshold, no symmetric metric fits. Direction-dependent cost calls for a quantile or pinball loss, where the asymmetry is a parameter. A threshold calls for treating the forecast as a score and evaluating the decision it triggers — which is classification, with a confusion matrix and its two costs.
The four questions
Before any metric is chosen, four properties of the cost of a miss decide it. Does the cost scale with the size of the thing predicted, or with the size of the miss alone? Does the cost of a miss grow faster than linearly with its size — is a large miss catastrophic, or just proportionally worse? Does the cost depend on the direction of the miss? Is there a threshold at which the cost jumps?
Each answer removes candidates. Scale-dependent costs remove percentage metrics. Linear costs remove squared error. Asymmetric costs remove every symmetric metric and point at a quantile loss. A threshold removes regression evaluation altogether and replaces it with a confusion matrix at the operating point.
How does the cost of being wrong behave?
when A minute late costs a minute of patience; ten units over costs ten units of holding.
cost MAE. The tail is accepted as a tail and must be monitored separately.
when A capacity breach or a safety margin: the twentieth unit of miss costs far more than the first.
cost RMSE, or a loss that grows faster still. The model will hedge toward the mean on skewed targets.
when Under-buying costs the intraday premium, over-buying costs the resale spread; the two are not equal.
cost Quantile (pinball) loss at the quantile the cost ratio implies. The asymmetry is a parameter that must be kept current.
when A penalty or an action triggers when the value exceeds a limit; the size of the miss elsewhere is nearly irrelevant.
cost Reframe as classification: the forecast is a score, the decision is binary, the metric is a confusion matrix with the two costs.
Asymmetric cost, written as a loss
When the two directions of a miss cost differently, the honest metric is a quantile loss. It charges a residual by a different slope on each side, and the ratio of the slopes is the ratio of the costs. Its minimiser is the quantile of the target at which the expected cost balances — which is exactly the purchase quantity the desk wants.
The loss is short to write and completely explicit about its opinion. That is its advantage over RMSE, which has an opinion too — that both directions cost the same and that the cost is convex — and never states it.
1import numpy as np2 3def pinball(y, f, tau):4 """tau in (0, 1): the quantile the forecast should target.5 A shortfall (y > f) costs tau per unit; a surplus (f > y) costs (1 - tau).6 """7 r = y - f8 return np.mean(np.where(r >= 0, tau * r, (tau - 1) * r))9 10# shortfall bought at an intraday premium ~3x the resale spread on a surplus11short_cost, surplus_cost = 3.0, 1.012tau = short_cost / (short_cost + surplus_cost) # 0.75: forecast the 75th percentile13 14# evaluating two candidates on the loss that matches the desk's cost15# a model that wins on MAE (tau = 0.5) can lose here by under-forecastingThe parameter tau is the whole metric choice. When the intraday premium changes, tau changes, and the model comparison has to be rerun — a bespoke metric is a dependency on the business, which is the point and the maintenance burden.
Classification in disguise
For the demand-response customers, the decision is whether consumption will cross a threshold. A regression model forecasts a number; the business acts on which side of the line it falls. The regression error is nearly irrelevant far from the line and decisive near it, and no residual-based metric captures that shape.
Treat the forecast as a score, apply the threshold, and evaluate the decision. The two costs are the penalty for a missed exceedance and the cost of an unnecessary intervention, and they are almost never equal — which is why accuracy would be the wrong number here too.
Illustrative counts. The regression model that produced these scores had a good MAE; most of its error was far from the threshold, where it cost nothing.
How to build it
Most important first.
- Write the cost of a miss as a function before choosing a metric. Ask: is a ten-unit miss on a hundred-unit customer the same as on a ten-unit customer? Are large misses catastrophic or merely bad? Does over cost the same as under? Is there a threshold at which the cost jumps? (Decision Before Model)
- Evaluate at the grain of the decision. If the purchase is the aggregate, evaluate the aggregate's error and bias; per-customer metrics are diagnostics for where the aggregate error comes from.
- For asymmetric costs, use a quantile loss whose asymmetry matches the ratio between the two costs, and report the realised cost on delayed actuals as the business metric (Business Metrics vs Model Metrics).
- For threshold decisions, reframe: the forecast is a score, the decision is binary, and the evaluation is a confusion matrix with the penalty as the false-negative cost (Prediction vs Decision, The Confusion Matrix).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Realised trading cost on the delayed actuals — the shortfall bought at the intraday price plus the surplus sold at the resale price — which is what the metric is standing in for.
- Aggregate bias per day, because per-customer errors that cancel are free and correlated errors are the whole cost.
- For the demand-response subset, the penalty rate at the operating threshold, not the regression error near it.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The cost function used to choose the metric is still the cost function the business faces — prices, penalties and thresholds have not moved since the metric was chosen.
- The grain at which the metric is computed is the grain at which the decision is made, and aggregating errors up to that grain does not hide a systematic per-segment bias.
- The residual distribution is such that the chosen loss's optimum — mean, median, quantile — is the quantity the decision needs; a metric chosen for symmetric residuals stops fitting once they skew.
- Offline: compute the realised cost function on the validation residuals for every candidate, alongside the standard metrics. If the model ranking under realised cost differs from the ranking under the chosen metric, the metric does not match the cost.
- Online: on delayed actuals, compute realised trading cost daily and compare it against the metric's prediction of it; a growing gap is either the cost function moving or the residual distribution moving.
- Over time: re-derive the metric from the current prices and thresholds quarterly, and rerun the model comparison. A ranking that changes is a metric that needed re-choosing.
What can go wrong
- The cost function is written once and the market changes it: the intraday premium doubles, the asymmetry flips, and the metric tuned for last year's prices is now steering the model toward the expensive side.
- The aggregate is evaluated and the per-customer errors are ignored, so a model that is right in total and badly wrong for a segment ships, and the segment's complaints arrive through a channel the metric does not watch (Evaluation Slices).
- The threshold customers are evaluated as regression because the pipeline was built for it, and the penalty rate is discovered from the invoice.
- A metric derived from the cost function is bespoke: harder to explain, not comparable to published results, and dependent on cost inputs someone must keep current.
- Evaluating at the decision grain hides per-entity behaviour, so a second evaluation at the entity grain is needed, and the two can disagree.
- Reframing a threshold decision as classification throws away the magnitude information a regression model had, which matters if the threshold later moves.
- "The metrics disagree, so the models are all about the same." They disagree because they weight errors differently. The disagreement is information about which errors each model makes, and the cost function decides which of those matters.
- "Use RMSE, it is the standard." It is the standard for a convex, symmetric, scale-dependent cost. If that is not the cost the business has, the standard is the wrong metric with a good reputation.
- "It is a regression problem because the target is a number." If the decision is a threshold on that number, the evaluation is a confusion matrix, and the regression error near the threshold tells you little about the penalty rate.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a metric is a loss over residuals and should match the cost of a miss holds for any regression task in any domain; the specific questions — scale, convexity, asymmetry, threshold — are the general ones.
- DOMAIN-SPECIFICEnergy trading has a market that prices the two directions of a miss explicitly, which makes the cost function unusually legible; in a domain like delivery-time estimation the cost is customer patience and has to be estimated before it can be encoded.
- CONTESTEDA serious position holds that bespoke cost-derived metrics are a trap: the cost function is never known precisely, it changes, and a model selected on a hand-built loss is fragile to that change, whereas a model that is strong on a standard metric like MAE is robust across cost functions and easier to reason about. The reply is that a standard metric is also a cost function, chosen implicitly, and the fragility is there either way — it is only visible when the choice is explicit.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Decision theory — the derivation of the optimal quantile from the ratio of the two costs is the newsvendor result, and the general statement that the evaluation metric should be the expected loss under the decision rule belongs to statistical decision theory.