Feature Freshness
Features update in seconds, minutes, hours or days. The model was trained on values of a particular age, and the serving architecture must deliver the same age or the model is reading a different signal.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How fresh do the features need to be, and does the serving path deliver the same freshness the training set had?
A delivery marketplace predicts whether an order will be late so it can warn the customer. The model uses restaurant load, courier availability and traffic. The product manager wants "real-time"; the data platform computes the features hourly; the model was trained on the hourly values.
Make everything real-time. Replace the hourly job with a streaming aggregation so the model always sees the latest numbers; fresher features can only help.
The model was trained on hourly values — orders_in_progress as of the top of the hour. Serving it per-second values changes the feature's distribution: the hourly value is a smoothed snapshot, the streaming value spikes with every burst. Same name, different signal.
- The model was trained on hourly values —
orders_in_progressas of the top of the hour. Serving it per-second values changes the feature's distribution: the hourly value is a smoothed snapshot, the streaming value spikes with every burst. Same name, different signal. - Warnings fire on transient spikes that the hourly-trained model never learned were transient. Conversion drops on orders that would have been on time.
- The opposite failure was also present before the change: some restaurants' aggregates were computed at minute 0 and read at minute 58, so the model saw hour-old load for orders placed during the dinner rush. Nobody noticed because training used the same stale value.
- The traffic provider's delay was invisible in training — the join used whatever value was stored — and in production the model treats a forty-minute-old index as current.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict at order time whether delivery will exceed the promised window. The label is the observed delivery time compared with the promise, known within an hour of ordering.
- The decision is whether to show a warning and possibly widen the promised window, which changes conversion — so a false alarm has a direct cost.
- One example is one order with restaurant-level and area-level aggregates: orders in progress, mean prep time, couriers idle, weather and traffic indices.
- Aggregates are computed by an hourly batch job from the order stream and written to a serving table. The training set joined each order to the aggregate row that was current for its hour.
- The traffic index comes from an external provider with a variable delay; sometimes the "current" value is forty minutes old.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Every feature has a freshness: the delay between the events it summarises and the moment it is read. A model trained on features of age
dlearned the relationship between *that* signal and the label. Serve features of aged'and the relationship is different — sometimes better, sometimes worse, never the one the weights encode. - Freshness is a property of the architecture, not of the feature. Batch computes and stores; serving reads the stored value some time later. The age at read time is the batch interval plus the wait since the last run plus any provider delay, and it varies across the interval.
- Required freshness comes from the decision: how quickly does the world change relative to the horizon? Courier availability changes in minutes; a restaurant's mean prep time changes in weeks. Serving both at one-second latency is expense without information; serving both hourly starves the first.
- The skew case: a feature computed daily but read at request time is a skew source *if training used end-of-day values*. Training saw the closed day; serving sees a partial day. Aligning the two — training on the value as it was readable, or serving the end-of-day value until the next close — removes the skew regardless of which freshness you pick.
Freshness is an architecture decision per feature
The product manager's "real-time" is a requirement on one or two features, not on the system. Courier availability changes in minutes and decides lateness; a restaurant's mean prep time changes over weeks and would be identical whether computed hourly or per second.
So the honest answer is a table: for each feature, how fast the underlying signal moves, what freshness the decision needs, and what the cheapest path that delivers it is. Most rows are batch. One or two justify a stream.
| Feature | Signal changes over | Freshness the decision needs | Cheapest path |
|---|---|---|---|
| couriers_idle_in_area | minutes | under a minute | streaming aggregate over courier status events |
| orders_in_progress (restaurant) | minutes | a few minutes | streaming count, or request-time query on a small table |
| mean_prep_time_7d | weeks | a day | daily batch |
| traffic_index | minutes, provider-limited | as fresh as the provider gives, with its age as an input | poll and store with timestamp |
| restaurant_rating | weeks | a day | daily batch |
The model was trained on values of a certain age
The freshness the model needs is the freshness it was trained with. That is the assumption a serving architecture has to hold, and it is checkable: log the timestamp of every feature value alongside the prediction and compare the age distribution with the one in the training set.
The trap in both directions is the same. Serve fresher than training and the model over-reacts to variation it learned to ignore; serve staler and it under-reacts to changes it learned to trust. Neither is caught by validation, which only ever saw one age.
Each feature reaches the model with the same age distribution — median and tail — that the training set was built with.
holds when Training joined on readable time, so it saw the batch lag; the batch schedule and streaming lag are unchanged since training; provider delays are within the range seen in training.
breaks when A pipeline is made faster or slower without retraining; a streaming job falls behind under load; a provider outage leaves a value frozen for hours while it still looks valid.
respond If the architecture changed, retrain on the new freshness. If the pipeline is behind, fall back to a defined stale-safe path (Serving Fallbacks) rather than serving a frozen value as current.
Daily feature, request-time read
The commonest freshness skew is not exotic. A feature is computed once a day over the closed day and stored. Training joins each example to "the value for that day" — the closed day's total. Serving reads the stored value at request time, which for most of the day is *yesterday's* closed total, and for a system that updates the row continuously is *today's partial* total.
Either way the serving value is a different quantity from the one training used. The fix is to decide which quantity the feature is and make both sides agree: train on the value that was readable at each example's time, and serve the same.
The training set joins each order to `orders_today` for its calendar date — the full-day total, known only at midnight. Serving reads a running count that is small in the morning and large at night.
The training set joins each order to the count as it stood at the order's timestamp; serving reads the same running count. Or: both sides use yesterday's closed total, refreshed once a day.
The model learns the relationship between the *quantity it will actually receive* and the label. A full-day total at 09:00 is information from the future during training and a fiction at serving time; a running count is a legitimate feature only if training saw it as a running count.
How to build it
Most important first.
- Decide freshness per feature from the rate at which the signal changes and the horizon of the decision, then build the cheapest architecture that meets it — batch for slow features, streaming for fast ones, request-time computation for the few that need it (Streaming Inference, Online Inference).
- Train on the freshness you will serve: join each training example to the feature value that was *readable* at the example's timestamp, including the batch lag, so the model learns the staleness it will see (Point-in-Time Correctness).
- Carry feature age as an explicit input where the delay varies — the traffic index with its timestamp — so the model can discount stale values instead of trusting them.
- Monitor feature age at the serving boundary and alert when it exceeds the training-time range; a stuck pipeline is a freshness failure the model cannot see (Data & Feature Tests).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-feature age at read time, as a distribution — median and tail — compared with the same distribution in the training set. Alignment between those two is the number that decides whether the model is reading what it learned.
- Warning precision on matured labels, split by feature-age bucket. If precision falls for the oldest bucket the freshness requirement is real; if it does not, the expensive streaming path is not earning its cost.
- End-to-end feature latency dashboards measure the pipeline, not the model. A fast pipeline serving a model trained on slow values is still skew.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The age distribution of each feature at serving time is within the range the training set was built with, per feature, including the tail.
- Batch-computed features are read with the same lag semantics training used — closed period or partial period — and a change to the batch schedule redeploys the model or retrains it.
- External provider delays stay within the bounds seen in training, and a delayed or missing value is passed to the model as a distinguishable state rather than as a stale number.
- Offline: for the training set, compute the per-feature age at readable time; compare against a week of logged serving ages. A shift in either direction is skew.
- Online: sample requests, record each feature's timestamp alongside the prediction, and evaluate matured labels by age bucket.
- Over time: alert on feature age per source; treat a stalled pipeline as a serving incident with a defined fallback, not as a data problem to fix tomorrow.
What can go wrong
- The streaming job falls behind under load; feature age grows from seconds to hours precisely during the dinner rush, when the model matters most, and the value looks valid.
- Freshness is improved for one feature and not its correlates: courier availability becomes real-time while restaurant load stays hourly, and the model, trained on both at the same age, sees a combination it never saw.
- The provider delay is fixed and the model — trained with delayed values — now receives fresher ones and over-weights them, because in training the fresh-looking index was always the delayed one.
- Real-time features need streaming infrastructure, state and its failure modes; hourly batch is cheap and reproducible. Paying for the first on features that do not change fast buys nothing.
- Training on readable-time values makes the offline number worse than training on idealised event-time values, because the model honestly learns with stale inputs. The worse number is the true one.
- Adding age as an input increases dimensionality and needs age to be logged in training, which most pipelines did not do until the problem appeared.
- "Fresher is always better." Fresher is *different*. A model trained on hourly snapshots has learned a smoothed signal; feed it per-second spikes and it fires on noise. Freshness has to match training or the model has to be retrained on the new freshness.
- "The feature is computed daily, so it is stale and useless." A daily feature that changes weekly is perfectly fresh. Staleness is age relative to the rate of change, not age alone.
- "Training used the same stale values, so it is fine." Only if serving is stale by the same amount at the same points in the cycle. A value computed at minute 0 and read across the hour is fresh at minute 1 and stale at minute 59, and training must have seen that spread too.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model learns the signal at the freshness it was trained on holds for any feature that summarises a moving world; only static attributes — a product category, a birth year — are exempt.
- DOMAIN-SPECIFICDelivery, ad auctions and fraud have signals that move in minutes and need streaming freshness for a few features; credit scoring and churn have signals that move in weeks, where daily batch is fully fresh and streaming is expense without information.
- SIMPLIFIEDThe lesson treats age as a single number per feature; real pipelines have different ages for different entities (a busy restaurant is refreshed more often than a quiet one), and any illustrative numbers are for the shape of the argument.
Where the depth lives
This domain teaches the model and hands the rest off by name.