Forecast Evaluation
Move the origin forward through time and score each forecast against what happened next; never shuffle. Report MAE or RMSE scaled against the naive forecast, per horizon and per segment — and treat MAPE with suspicion near zero.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How do you produce a forecast error that predicts the error the forecast will have in production, and which error metric maps to the decision?
An energy retailer built a model to forecast hourly household consumption for a day-ahead purchase. Cross-validated error was excellent, the model was deployed, and the first month's actual error was roughly three times the reported number. The procurement desk over-bought in the cold snap and under-bought the warm week after.
Cross-validate. Five folds, shuffle the rows, average the error. It is the standard protocol, it uses all the data for evaluation, and the variance across folds gives an uncertainty.
A shuffled fold holds out 2 p.m. on a Tuesday and trains on 1 p.m. and 3 p.m. the same day. The model interpolates between its neighbours and the error is a fraction of any real day-ahead error (Temporal Leakage). The cross-validated number measures interpolation, not forecasting.
- A shuffled fold holds out 2 p.m. on a Tuesday and trains on 1 p.m. and 3 p.m. the same day. The model interpolates between its neighbours and the error is a fraction of any real day-ahead error (Temporal Leakage). The cross-validated number measures interpolation, not forecasting.
- The features were weather actuals. At noon the next day's weather is a forecast, which is wrong by a few degrees on a bad day; the model was evaluated with the answer and deployed with a guess (Point-in-Time Correctness).
- The error was reported as a single MAPE. Overnight hours have small consumption and the percentage error there is huge and noisy, dominating the average, while the expensive daytime hours are drowned out (MAPE and Its Caveats).
- The evaluation pooled all hours and all seasons. The model's error in a cold snap — the hours where the purchase cost is highest — was never seen separately, and it was the worst (Evaluation Slices).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict per-hour aggregate consumption for the next day, at the day-ahead origin (noon). The label is the metered total, available two days later.
- The decision is a purchase quantity per hour; over-buying sells back at a loss and under-buying buys on the spot market at a premium, so the cost is asymmetric and per hour.
- Three years of hourly consumption, weather actuals, and weather forecasts as they were issued at noon each day — the last of which is the covariate the model may use, since the actual weather is not known at the origin.
- The original evaluation used weather actuals as features and a shuffled five-fold cross-validation over hourly rows.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Rolling-origin evaluation (forward validation) picks a sequence of origins
t₁ < t₂ < …, and for each fits on data up totᵢ, forecaststᵢ+1..tᵢ+Husing only covariates known attᵢ, and scores against what happened. The origins advance; the training window can grow or slide. Every score is a real forecast made without the future, so the average is an estimate of production error. A random split is not a degraded version of this; it measures a different quantity (Time-Series Validation). - MAE is the mean absolute error, in the series' units, robust to outliers, and it is the right number when the cost of an error is proportional to its size. RMSE squares before averaging and so weights large errors more; it matches a cost that grows faster than linearly, and it is what a model trained on squared loss is optimising (MSE, RMSE and MAE). MAPE divides by the actual: it is undefined at zero, explodes near zero, and penalises over-forecasts more than under-forecasts of the same size, because the denominator is the actual and not the forecast. Scaled errors divide the model's MAE by the naive forecast's MAE on the training data, giving a unitless number comparable across series where one means "as good as naive".
- Error is not one number. It varies with horizon (later is harder), with segment (cold snaps, holidays, night), and with the regime the origin sits in. A pooled average hides all three, and the decision reads one cell of that table at a time.
Why the origin has to move forward
A forecast is made at a moment, from what was known at that moment, about what came after. An evaluation that does not reproduce that — that trains on hours after the held-out hour, or uses the actual weather instead of the forecast issued at the origin — is evaluating a model with access to the future. It will look better than any deployed model can, by an amount nobody can compute from the number itself.
Rolling origin reproduces the moment. Fit to noon on day one, forecast day two with day one's noon weather forecast, score against day two's meter. Advance to noon on day two. Every score is a genuine forecast, and the average over a year of them is what the desk will see.
1def rolling_origin(series, weather_issued, origins, horizon_hours, fit, predict):2 """series: hourly y with ts. weather_issued: rows (issued_at, target_ts, temp_fc).3 At each origin, only forecasts issued at or before the origin are visible."""4 scores = []5 for t in origins: # every noon in the eval year6 train = series[series.ts <= t]7 model = fit(train)8 targets = series[(series.ts > t) & (series.ts <= t + horizon_hours)]9 wx = weather_issued[(weather_issued.issued_at <= t)] # as-of the origin10 wx = wx.sort_values("issued_at").groupby("target_ts").last() # latest visible issue11 X = build_features(train, targets.ts, wx) # lags end at t; covariates as-of t12 yhat = predict(model, X)13 naive = seasonal_naive_block(train, targets.ts, m=24 * 7)14 for h, (y, f, n) in enumerate(zip(targets.y, yhat, naive), start=1):15 scores.append({"origin": t, "h": h, "err": abs(y - f), "naive_err": abs(y - n),16 "segment": segment_of(targets.ts.iloc[h - 1], wx)})17 return scores # aggregate by h and by segment; scale err by training-period naive MAEThe weather_issued filter is the line that makes the evaluation honest, and it needs a table that keeps every issue with its issued_at. A weather table that stores only the latest forecast per target hour has already revised the past, and no evaluation over it can be trusted.
Which error, and against what
MAE says how far off the forecast is, on average, in kilowatt-hours. RMSE says the same but punishes the big misses harder, which matches a cost that spikes when the spot market is expensive. MAPE says how far off in percent — and at 4 a.m. when consumption is small, a modest absolute miss is a large percentage, so the metric is dominated by the hours where the money is smallest. Scaled error says how the model compares to last week's same hour, which is the question the decision actually turns on.
None of them is the purchase cost. The desk pays a different price per hour for over- and under-buying, and that number, computed on the rolling-origin forecasts, is the one that says whether the model is worth its cost. The error metrics are the diagnostics that explain it.
| Metric | Units | Weights large errors | Near-zero actuals | Symmetric | Comparable across series | Use when |
|---|---|---|---|---|---|---|
| MAE | Series units | Linearly | Fine | Yes | No | Cost is proportional to error size |
| RMSE | Series units | Quadratically | Fine | Yes | No | Big misses cost disproportionately; model trained on squared loss |
| MAPE | Percent | By 1/actual | Undefined at zero, explodes near it | No — over-forecasts penalised more | Nominally, misleadingly | Series far from zero where the business speaks in percent |
| Scaled (MASE-style) | Ratio to naive MAE | Linearly | Fine | Yes | Yes | Comparing to the baseline and across series of different scale |
| Decision cost | Money | As the cost does | As the cost does | As the cost does | Yes | Deciding whether to deploy; the others explain this one |
One number, many conditions
The pooled year-long error was fine. The cold-snap error, had it been reported, was several times larger, and it was the only week the desk lost real money. A forecast's error is a distribution over horizons, seasons and regimes, and the decision reads the expensive corner of it. Evaluation has to report that corner, or the reported number is an average that never occurs.
This is the gap that closed the month badly: offline said one thing because it pooled interpolated errors with actual weather; production said another because it forecast with noon's weather guess through a cold snap the evaluation year never contained.
Shuffled five-fold cross-validation, weather actuals as features, one pooled MAPE across all hours: excellent, and stable across folds.
Per-hour error against metered actuals roughly three times the reported figure on ordinary days, far worse in the cold snap; the desk over-bought the snap and under-bought the warm week after.
- 1Shuffled folds let the model interpolate between neighbouring hours, so the offline number measured interpolation, not day-ahead forecasting.
- 2The evaluation used actual weather; production has the noon forecast, which missed the snap's depth by several degrees, and the model had never been scored under that error.
- 3The pooled MAPE was dominated by low-consumption overnight hours and hid the daytime error where the purchase cost concentrates; the evaluation year also contained no comparable cold snap.
How to build it
Most important first.
- Evaluate only by rolling origin, with origins spaced as production will run them (every noon), over a period that contains the conditions the decision cares about — at least a year, so every season appears.
- At each origin, use the covariates as they were known at the origin: the weather forecast issued at noon, not the actual. Store forecasts as issued with an as-of timestamp; a covariate table without one cannot be evaluated honestly (Feature Freshness).
- Report MAE and RMSE in units, scaled against the seasonal naive, per horizon and per segment. Use MAPE only for series bounded well away from zero, and say so.
- Report the metric the decision pays: the purchase cost of the forecast errors under the actual buy-back and spot prices, per hour, summed. This is the number the desk decides on (Business Metrics vs Model Metrics).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Rolling-origin MAE scaled against seasonal naive, per horizon and per segment, over the last year of origins. The scaled number says whether the model beats the baseline; the segments say where.
- The purchase-cost metric on the same rolling forecasts. This is what maps to the decision; the error metrics are proxies.
- Do not measure shuffled cross-validation error, pooled MAPE, or anything computed with covariates the origin did not have. Each of those is a number about a model that will not exist in production.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The covariates available at the serving origin are the same ones the rolling-origin evaluation used, at the same as-of time — the noon weather forecast, not a later revision.
- The evaluation period contained the conditions the deployment period will have; if a season or a regime was absent, the reported error does not cover it and the interval should say so.
- The naive baseline used for scaling is computed the same way in the serving-time report as in the evaluation, so the scaled number means the same thing week to week.
- Offline: replay a year of noons through the full pipeline — features as of noon, forecast, score against the metered actual — and compare per-segment error to the deployment expectation. This is the only offline number that predicts production.
- Online: score each day's forecast when the meter data arrives two days later, per hour, and compare to the offline rolling-origin distribution; a persistent gap means the serving-time covariates differ from the evaluation's (Train / Serve Skew).
- Over time: track scaled error per week and per segment; a rising number in one segment with a flat number elsewhere is a regime change in that segment, and the response is to look, not to retrain (Drift Is Not Failure).
What can go wrong
- The rolling-origin evaluation is implemented correctly for the target but the weather-forecast table is rebuilt nightly with revised forecasts, so "as issued at noon" is silently the last revision (Backfills).
- The evaluation period is the last year, which contained no cold snap; the model's error in the one condition that dominates purchase cost was never measured.
- The scaled error is computed against the naive forecast's error on the *test* period rather than the training period, and the scale moves with the period, so the number is not comparable across weeks.
- Rolling-origin evaluation is many fits instead of one, and on a slow model it is expensive; it is also the only protocol that produces a forecast error.
- Storing covariates as issued, with an as-of dimension, is more storage and a data-engineering discipline; without it the evaluation is quietly using revised data.
- A per-horizon, per-segment table is harder to read than one number, and the decision needs the table.
- "Random split works for everything — cross-validation is the gold standard." On a series a shuffled fold lets the model interpolate between adjacent hours. The number is a fact about interpolation and says nothing about tomorrow.
- "MAPE is best because it is unitless and comparable across series." It is undefined at zero, explodes near it, and is asymmetric; on a series that touches zero overnight it is dominated by the hours that cost least. Scaled error is unitless and has none of those problems.
- "The rolling-origin error is worse than the cross-validated one, so the rolling-origin implementation must be wrong." It is worse because it is a forecast error. The cross-validated number was the one that was wrong.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a shuffled split on a series measures interpolation rather than forecasting, and that covariates must be as-of the origin, follow from what a forecast is; they apply to every series, horizon and model.
- TASK-SPECIFICWhich error metric maps to the decision depends on the cost shape: MAE for linear cost, RMSE for cost that grows with error size, a pinball or quantile loss when the decision is a quantity under asymmetric cost; MAPE fits only series far from zero where relative error is what the business speaks.
- CONTESTEDSome practitioners hold that blocked cross-validation — contiguous folds without shuffling, trained on data both before and after the held-out block — is an acceptable and more data-efficient alternative to rolling origin for stationary series. That is defensible for a stationary series with no trend; the reply is that most business series are not stationary, that training on the future of a block leaks the regime, and that the data efficiency gained is paid for in optimism nobody can size.
Where the depth lives
This domain teaches the model and hands the rest off by name.