Forecasting
The target is a future value of the series you already have. Features are lags and windows that end at the forecast origin, the naive forecast is the baseline, and beating it is harder than it looks.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
When the thing to predict is the same series at a later time, what are the features, what is the baseline, and what does a model have to beat to be worth deploying?
A grocery chain wants next week's demand for each product in each store so that orders go out on Thursday. Today a planner types a number based on last week and gut feel. The team has three years of daily sales and wants a model to replace the gut feel.
Treat it as regression. Build features from the history — last week's sales, the rolling mean, the day of week — fit a gradient-boosted model, evaluate on a held-out set. It is a table with a numeric target.
The rolling mean was computed over a centred window, so for a Monday it includes the following Tuesday to Thursday. Offline the feature is excellent; on Thursday the following days do not exist (Temporal Features).
- The rolling mean was computed over a centred window, so for a Monday it includes the following Tuesday to Thursday. Offline the feature is excellent; on Thursday the following days do not exist (Temporal Features).
- The held-out set was a random 20% of rows. A held-out Wednesday sits between a training Tuesday and a training Thursday from the same series; the model interpolates and the metric is far better than any real forecast will be (Temporal Leakage).
- Nobody ran the seasonal naive forecast — same day last week — as a baseline. When someone does, it is within a few percent of the model on most series, and better on the intermittent ones (Baselines Are Mandatory).
- The model was trained on three years including a period when the chain changed its promotion policy. It forecasts the average of two regimes, which was never the right answer in either.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict units sold per store-product for each of the seven days after the forecast origin (Thursday). The label is the realised sales figure from the point-of-sale feed.
- Sales are censored by stock-outs — a day with zero stock records zero sales regardless of demand — so the label is sales, not demand, and the model will learn to forecast what the store could sell (Label Construction).
- One example is one store-product-day: the sales figure, price, promotion flag, day of week, and whatever aggregates can be computed from the series' own history up to the forecast origin.
- Three years of daily rows for around forty thousand store-product series. Many are intermittent — most days zero — and a few hundred high-volume series carry most of the revenue.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A forecast is a prediction of
y[t+h]from information available at timet, the forecast origin. Every feature must be a function of the series and its covariates up to and includingt: a lagy[t-k], a window statistic overy[t-w..t], a calendar feature oft+h, a known-in-advance covariate such as a planned promotion. Anything that reads pasttis a leak, however innocent it looks (Point-in-Time Correctness). - The naive forecast
ŷ[t+h] = y[t]and the seasonal naiveŷ[t+h] = y[t+h-m](last week's same day,m = 7) are the baselines. They encode the two strongest facts about most series — persistence and seasonality — with no parameters. A model that cannot beat them is adding noise. - Three model families compete. Classical models — exponential smoothing and ARIMA — fit each series separately with a few parameters that model level, trend and seasonality explicitly; they are cheap, robust on short series and interpretable. A gradient-boosted model on lag features fits one model across all series, can use covariates and cross-series structure, and needs the features to be built correctly. Sequence networks learn the features from raw windows and can beat both on long series with rich covariates, at a cost in data, tuning and opacity (Sequence Models).
Every feature has an origin
A lag is a value from before the origin. A rolling mean is an average of values before the origin. A calendar feature is a fact about the target date that was known before the origin. A planned promotion is a covariate known before the origin. The word that repeats is the constraint, and the bug is always a feature that violates it quietly — a centred window, a "last seven days" that includes today, a promotion flag backfilled from what actually happened.
In code the origin is an argument. A feature function that takes the whole frame and a row index can read anything; a feature function that takes the frame and an origin t and filters to timestamp <= t cannot leak by accident. The second is slower and correct.
1def features_at(series, t, horizon):2 """series: DataFrame with columns [ts, y, promo_planned]; one store-product.3 Everything below reads rows with ts <= t, and calendar facts about t + h."""4 hist = series[series.ts <= t] # the constraint, in one place5 y = hist.y.values6 feats = {7 "lag_1": y[-1],8 "lag_7": y[-7],9 "mean_7": y[-7:].mean(),10 "mean_28": y[-28:].mean(),11 "zeros_28": (y[-28:] == 0).mean(), # intermittency12 }13 target_day = t + horizon14 feats["dow"] = target_day.dayofweek # known in advance15 # a promotion is a feature only if it was *planned* before t, not backfilled16 feats["promo"] = series.loc[series.ts == target_day, "promo_planned"].item()17 return feats18 19# the seasonal-naive baseline, computed with the same origin20def seasonal_naive(series, t, horizon, m=7):21 hist = series[series.ts <= t]22 return hist.y.values[-m + (horizon - 1) % m - 1]The promotion column is the subtle one. promo_planned must be the plan as it stood at t, which is a slowly-changing record; a column that holds what actually ran is a leak dressed as a covariate.
What the model has to beat
Last week's same day is a forecast. It has no parameters, needs no pipeline, adapts instantly to a regime change, and on many series it is within a few percent of anything a model produces. On intermittent series — most days zero — it is often better, because a model regresses toward a small positive mean and is wrong every day.
So the number that matters is the model's error divided by the naive error, per series segment. A ratio near one on the long tail means the model should not be deployed there; a ratio well below one on the high-volume head means it should. The pooled number answers neither question.
One MAE across forty thousand series, dominated by the few hundred high-volume ones, with the intermittent tail contributing almost nothing to the number and losing to seasonal naive throughout.
Model MAE over seasonal-naive MAE, reported separately for high-volume, mid-volume and intermittent series and per horizon. The head shows a clear win; the tail shows a loss; the deployment is per segment.
The decision is "deploy the model for this series or keep the naive forecast", and it is made per series. A pooled number cannot make that decision, and scaling by the naive error makes series of different volume comparable at all.
The origin must be the same in production
Training computed features at Thursday close. Production runs Thursday morning, and the latest row is Wednesday — or Wednesday's partial day if the feed is mid-load. Every lag is shifted by a day and every window is short by one. The model was never trained on that, and nothing in the artifact can tell.
The assumption that has to hold after deployment is about the origin: that the serving path and the training path agree on what t is and on which rows exist at t. It is the forecasting-specific instance of train/serve skew, and it is the most common bug in a forecasting system.
At serving time the features are computed from exactly the rows with timestamp at or before the forecast origin as training defined it, and the point-of-sale feed for those rows has fully arrived.
holds when The serving job runs after the feed's completeness check for the origin day; the feature function takes the origin explicitly and is the same function training used; late-arriving stores are excluded rather than treated as zero.
breaks when The job is moved earlier to meet the order deadline; the feed is late for a region; a partial day is loaded and counted; a store's clock is wrong and its rows land on the wrong day.
lag_1 is zero while its mean_28 is not.respond Do not retrain. Fix the origin — move the job, wait for completeness, or exclude late series and fall back to seasonal naive for them.
How to build it
Most important first.
- Build features with an explicit origin: every lag and window is computed from rows with timestamp
<= t, in a function that takestas an argument. A feature that cannot name its origin is a leak waiting to happen. - Run the naive and seasonal naive forecasts first and report every model relative to them. The question is not "what is the error" but "how much of the naive error did the model remove" (Beating the Baseline).
- Evaluate with a rolling origin: fit up to
t, forecastt+1..t+h, advancet, repeat (Forecast Evaluation). Never a random split. - Choose the family from the data: short intermittent series favour classical or even naive per series; many related series with covariates favour a global boosted model; long dense series with rich context are where a sequence network can earn its cost. Often the answer is different per segment.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Scaled error against the seasonal naive, per horizon and per series segment — high-volume, intermittent, new. This is the number that says whether the model is worth its cost, and where.
- The order-cost metric downstream — units over-ordered times holding cost plus units short times stock-out cost — on the rolling-origin forecasts. This is the number the business decides on.
- Do not measure pooled MAPE across all series. The intermittent ones have zeros in the denominator and the high-volume ones vanish in the average (MAPE and Its Caveats).
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The serving path computes every lag and window from data timestamped at or before the forecast origin, with the same origin definition training used — Thursday close, not Thursday morning.
- The relationship between recent history and the next week — the persistence and seasonality the model learned — continues to hold; a change in promotion policy, assortment or store hours breaks it (Concept Drift).
- The point-of-sale feed arrives complete before the forecast is made; a late-arriving store's data makes its lags look like a collapse in demand.
- Offline: rolling-origin evaluation over the last year of Thursdays, model against seasonal naive, per horizon and per segment. A model that does not win on a segment is not deployed for that segment.
- Online: for the first month, log the feature vector at forecast time and recompute it from the warehouse a week later with the same origin; a mismatch is a skew, and it is the most common forecasting bug.
- Over time: track scaled error against seasonal naive per week. The naive forecast adapts to a regime change automatically; a widening gap in the wrong direction is the model failing to.
What can go wrong
- The feature pipeline in production computes the rolling window from the latest available data, which on Thursday morning is Wednesday's partial day — a different origin from training, and the feature is systematically low (Train / Serve Skew).
- A promotion planned for next week is a legitimate known-in-advance covariate; the promotion flag in the training table was backfilled from what actually ran, including last-minute promotions no forecast could have known (Feature Freshness).
- The model beats seasonal naive on the high-volume series and loses on the intermittent ones; deployed for all of them, it makes the long tail worse and the pooled metric hides it (Evaluation Slices).
- Per-origin feature construction is slower and more code than a rolling window over a sorted frame, and it is the difference between a forecast and a leak.
- A global model across forty thousand series is one artifact to serve and monitor and one set of hyperparameters to tune; it also means one bad feature affects every series at once.
- Reporting against the naive baseline makes many models look modest, which is honest and unwelcome.
- "It's just regression with a time column." The time column is the whole problem: it decides which rows may be features, which may be training data, and which may be evaluation. Regression tooling ignores all three.
- "Random split works for everything — we shuffled and the model generalises." A shuffled split on a series lets the model interpolate between neighbours it will never have at forecast time. The number is not a forecast error.
- "The model has lower error than last week's value, so ship it." Lower on which series, at which horizon, and by how much relative to the noise? Seasonal naive is often within a few percent, and on intermittent series it often wins.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat features must end at the forecast origin and that the naive forecast is the baseline follow from the definition of a forecast, and apply to every series and model family.
- DATA-SPECIFICWhich model family wins depends on the series: on short or intermittent series a per-series classical model or seasonal naive is usually best; on many related series with known covariates a global boosted model; sequence networks need long, dense series with rich context to justify their data and tuning cost.
- CONTESTEDA respected position holds that global deep models trained across many series now beat classical and boosted approaches broadly enough that per-series classical modelling is obsolete outside tiny datasets. That is supported by several public competitions on dense retail data; the reply is that those wins are on the segments with the most data, that they vanish on intermittent and short series, and that a per-series exponential smoothing model needs no feature pipeline to leak.
Where the depth lives
This domain teaches the model and hands the rest off by name.