FeaturesGENERALTASK-SPECIFICCONTESTED

Temporal Features

Windows, lags, recency, calendar features and "as of" timestamps. Every one is anchored to a clock, and the rule is that the anchor is the prediction time and no window ends after it.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Time is the axis along which features leak, skew and go stale. How do you build features from it that mean the same thing in training and serving?

The problem

A utility forecasts hourly electricity demand per substation a day ahead. The features are lags, rolling means, hour of day, day of week and holidays. Validation was excellent. In production the day-ahead forecast is markedly worse than validation promised, and worst on the days after a public holiday.

The obvious approach

Lags and rolling windows are the standard forecasting features; compute them for every hour relative to that hour. Add calendar features. Validate on a later period.

Why it breaks

The one-hour and twenty-four-hour lags were computed relative to the target hour. At noon today, forecasting tomorrow at 3pm, the "one hour ago" value is tomorrow at 2pm — unknown. In training it was known, and it is the strongest feature. Serving substitutes the latest available value and the model, trained on the true lag, is badly off.

How it breaks — usually after the offline metric looked fine

The one-hour and twenty-four-hour lags were computed relative to the target hour. At noon today, forecasting tomorrow at 3pm, the "one hour ago" value is tomorrow at 2pm — unknown. In training it was known, and it is the strongest feature. Serving substitutes the latest available value and the model, trained on the true lag, is badly off.
The rolling 24-hour mean was computed over the 24 hours ending at the target hour, so it included hours after the forecast was issued. Same leak, wider window.
The holiday flag was joined on the UTC date, so for an evening-shifted timezone the flag applied to the wrong hours, and the day after a holiday — when demand rebounds — was partly flagged as the holiday itself. The model learned a smeared holiday effect and cannot reproduce the rebound.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict demand for each hour of tomorrow, at noon today, per substation. The label is metered demand for that hour.
The forecast is used to schedule generation, so the number that matters is error at the horizon actually forecast — twelve to thirty-six hours ahead — not one hour ahead.

Data

One example is one substation-hour with lags of demand at 1, 24 and 168 hours, rolling means over 24 hours and 7 days, and calendar features.
Training features were built from the full demand series, with lags relative to the target hour. The one-hour lag of tomorrow's 3pm is tomorrow's 2pm, which at noon today does not exist.
Holidays came from a calendar table joined by date, in local time; the demand series was in UTC.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A temporal feature is a function of the series at times relative to an anchor. The anchor has to be the prediction time — when the forecast is issued — not the target time. A lag of h hours for a forecast issued at t0 for target t is the value at t0 - h, or equivalently at t - horizon - h; a lag computed at t - h is only legitimate if h >= horizon (The Forecast Horizon).
A window is defined by its end, and the end must be at or before the anchor: [t0 - w, t0). Any window ending after t0 contains information the forecast could not have had. This is the temporal-leakage rule (Temporal Leakage) restated for features, and the day-ahead horizon makes it bite harder than a nowcast would.
Recency features — time since the last event — depend on what "last" means as of the anchor, not as of the label. Calendar features depend on the timezone and on the calendar version: a holiday table is reference data that changes, and local versus UTC is a join key mismatch that produces a shifted effect rather than an error.
Serving lag adds a second clock. If the meter data reaches the serving system with a two-hour delay, the latest value available at noon is 10am, and the training feature must be built as-of t0 - 2h to match (Feature Freshness).

The anchor is the issue time

The training set had a row per substation-hour, and the lag features were computed relative to that hour: the demand one hour, one day and one week before. That is a fine feature set for a one-hour-ahead nowcast. The product is a day-ahead forecast issued at noon. For tomorrow at 3pm, the one-hour lag is tomorrow at 2pm, which is three hours *after* the forecast is due and twenty-six hours after it is issued.

Re-anchoring changes every feature: the "latest" demand is noon today minus the data lag, the one-day lag is yesterday at the target hour, and the horizon — twelve to thirty-six hours — becomes a feature in its own right. The honest model is worse than the leaked one, and it is the only one that forecasts.

leakagedemand_lag_1hA lag anchored to the target

looks like The single most predictive feature in a demand model: last hour's demand. Present in training for every row.

why it leaks It is computed relative to the target hour. For a day-ahead forecast the target hour's previous hour is in the future at issue time. The training feature holds a value the forecast could never have.

offline

Validation error is tiny, even on a later period, because the leak is inside each row and a time split does not touch it.

production

The serving path has no value for tomorrow at 2pm and substitutes the latest reading. The model, trained to trust a one-hour lag, is applied to a twenty-six-hour lag, and the forecast is badly off — worst where demand changes fastest, such as the rebound after a holiday.

fix Anchor every lag and window to the issue time t0 minus the serving lag; add the horizon as a feature; assert lag_time <= t0 - lag in the pipeline.

when this feature is fine The same feature is exactly right for a nowcast issued each hour for the next hour, where the target's previous hour is the issue time itself — and a 24-hour lag is fine for any horizon up to 24 hours, because it is always in the past at issue time.

Lags and windows anchored to the issue time

1def temporal_features(series, t0, t, serving_lag):
2    """series: hourly demand indexed by UTC timestamp. t0: issue time.
3    t: target hour. Everything is relative to what was KNOWN at t0."""
4    known_until = t0 - serving_lag              # latest reading serving will have
5    hist = series[series.index < known_until]    # strict: nothing at/after
6    horizon_h = (t - t0) / pd.Timedelta("1h")
7    return {
8        "horizon_h": horizon_h,
9        "latest": hist.iloc[-1],                                   # not t - 1h
10        "lag_24h_of_target": hist.get(t - pd.Timedelta("24h")),    # None if in future
11        "lag_168h_of_target": hist.get(t - pd.Timedelta("168h")),
12        "mean_24h": hist[hist.index >= known_until - pd.Timedelta("24h")].mean(),
13        "hour_local": t.tz_convert(LOCAL_TZ).hour,
14        "dow_local": t.tz_convert(LOCAL_TZ).dayofweek,
15        "holiday": is_holiday(t.tz_convert(LOCAL_TZ).date(), CALENDAR_VERSION),
16    }

The lag_24h_of_target line returns None when the horizon exceeds 24 hours — a feature that is sometimes unavailable by construction, which the model must be trained with a missingness policy for (Missing Data) rather than a substituted value.

Calendars are reference data in a timezone

Hour of day, day of week and holiday flags look like the safest features in the set: no window, no lag, no fitted state. They have a timezone and a version. The demand series was in UTC; the holiday table was joined by UTC date; for a timezone several hours off UTC, the holiday flag covered part of the previous or following local day. The model learned a holiday effect smeared across a day boundary, and the sharp post-holiday rebound was partly labelled as holiday.

The fix is to compute every calendar feature in the entity's local timezone, from a versioned calendar table whose version ships with the model, and to treat a calendar update like any other upstream change: the serving path must not pick up a new holiday the model has never seen without a decision about it.

Calendar and clock failures

Trigger	Symptom	Cause	Response
Holiday table joined on UTC date	Holiday effect appears shifted; post-holiday hours partially flagged	Local date and UTC date differ for part of the day	Convert to local time before deriving any calendar feature; test on a timezone far from UTC
Daylight-saving transition	One hour duplicated or missing per year; lag alignment off by one on those days	Hour-of-day computed in local time, lags computed in UTC	Keep the series in UTC for lags; derive hour-of-day in local time; treat the transition day as its own flag
Calendar table updated with a new holiday	Serving flags a date the model never saw flagged	Reference data changed without a model change	Version the calendar with the artifact; alert on a flag value that never occurred in training
Metering lag grows after an upgrade	Forecast error rises at short horizons; "latest" feature is staler than training assumed	Training built features as-of `t0 - 2h`; serving now has `t0 - 6h`	Log the observed lag; rebuild training features at the new lag; treat it as a serving-freshness change

Validate at the horizon you serve

A pooled error over all hours hides that the model is excellent one hour ahead and poor thirty hours ahead. The scheduler only uses the thirty-hour number. Forecast-origin validation — for each issue time in the holdout, score every horizon — reports the error the product actually delivers, and lets a seasonal-naive baseline be compared at each horizon, which is the comparison that decides whether the model earns its cost.

The assumption under all of this is that the horizons, the lag and the calendar in production are the ones training assumed. Each is an operational parameter that can change without anyone editing the model.

must stay trueSame anchor, same lag, same calendar

Every temporal feature is anchored to the issue time with the serving lag applied, calendar features use the local timezone and the shipped calendar version, and production horizons match those validated.

holds when Training rows carry t0 and t; the pipeline asserts window and lag bounds; serving lag is logged and matches the training parameter; the calendar version is in the artifact.

breaks when A library default re-aligns a rolling window on the target; the metering lag changes; the calendar is updated upstream; the product starts issuing forecasts at a different time of day, changing every horizon.

how you would know The truncated-series rebuild test; error by horizon against validation as actuals arrive; the logged data lag at issue time; an alert on calendar flags never seen in training.

respond Fix the anchor or the parameter, rebuild the training features at the observed lag, and re-validate by horizon before retraining anything.

How to build it

Most important first.

Fix the anchor: every training row carries an issue time t0 and a target time t; every lag and window is computed relative to t0, and the horizon t - t0 is itself a feature.
Assert window_end <= t0 and lag_time <= t0 - serving_lag in the feature pipeline tests; make the serving lag an explicit parameter of training feature construction.
Compute calendar features in the local timezone of the entity, from a versioned calendar table, and join on local date; ship the calendar version with the model.
Validate by forecast origin: for each issue time, score all horizons, and report error by horizon rather than pooled (Time-Series Validation).

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Error by horizon on a later-period holdout with features built as of the issue time. This is the number the generation scheduler experiences; a pooled or one-hour-ahead number is not.
Error on the day after a holiday, specifically, since the smeared calendar effect concentrates there.
The validation number computed with target-anchored lags is not a forecast metric at all.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Every lag and window is anchored to the issue time, ends at or before it minus the serving lag, and the serving lag assumed in training matches the observed lag in production.
Calendar and holiday features are computed in the same timezone from the same calendar version in both paths, and the version ships with the artifact.
The horizon distribution in production matches the horizons the model was trained and validated on.

How to verify — offline, online, and over time

Offline: for a sample of training rows, rebuild the features with the series truncated at t0 - serving_lag and assert nothing changes.
Offline: score by horizon on forecast-origin validation; compare against a seasonal-naive baseline at each horizon.
Online: log the actual data lag at each issue time and the features used; compare error by horizon and by calendar flag against validation as actuals arrive.

What can go wrong

Failure modes in production

Lags are re-anchored to t0, but the rolling mean is still computed by a library default that centres or right-aligns the window on the target hour.
The serving lag is modelled as two hours, and after a metering system upgrade it becomes six; training features assume freshness the serving path no longer has, silently.
The calendar table is updated with a new public holiday, the serving path picks it up, and the model — trained on the old calendar — has never seen that flag on that date.

What the recommended approach costs

Anchoring to the issue time discards the most recent, most predictive lags for long horizons, and the honest number is much worse than the target-anchored one.
Forecast-origin validation with per-horizon reporting is many more evaluations than a pooled score, and needs a rolling scheme to be stable.
Modelling serving lag explicitly couples training feature construction to an operational parameter that has to be monitored.

Misreads

"Lag features are safe because they only look backward." Backward from the target hour is forward from the issue time whenever the horizon exceeds the lag. Backward from the issue time is the only safe direction.
"We validated on a later period, so temporal leakage is impossible." The split is honest; the features are not. Target-anchored lags leak inside every row regardless of the split.
"Calendar features are static, so they cannot skew." They are reference data in a timezone. A UTC join or a calendar update changes them without anyone touching the model.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALAnchoring to the prediction time and ending every window at or before it holds for any model consuming time-derived features — forecasts, churn, fraud, maintenance — regardless of model family.
TASK-SPECIFICFor a multi-horizon forecast the anchor and target differ by the horizon and the mismatch is the main leak; for a classification with a single prediction time per row, the anchor and the row time coincide and the rule reduces to "no window past the row time".
CONTESTEDSome forecasters argue that training on target-anchored lags with a separate model per horizon, then feeding recursive predictions for the unknown lags at serving, is a legitimate and often stronger strategy than direct issue-time features. That is a real method with real results; the objection is that the recursive version must be validated as it will be served — with predicted, not actual, lags — and that the target-anchored validation number is never that.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringevent-time processing-time late-arriving-data windows

Databasewindow-functions