Feature Engineering
A feature is a transformation from raw records to a number the model can use. It is learned from training data, and it has to be reproduced identically at serving time — which is where it usually breaks.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model is a function of its features, not of the raw data. What is a feature, who computes it, when, and what must be true for the same feature to exist in production?
A marketplace wants to predict which new listings will sell within a week so it can promote the ones that will not. A data scientist built forty features in a notebook from the listing table, the seller table and the event log. The model is good offline. The engineering team now has to make those forty features exist at the moment a listing is created.
Feature engineering is the creative part: derive as many informative columns as possible from the raw data, keep the ones that help, hand the list to engineering to reproduce.
Half the features cannot be computed at serving time as defined: the category statistics include listings created after this one, the seller history includes this listing's own outcome, the vectoriser vocabulary includes words from future titles.
- Half the features cannot be computed at serving time as defined: the category statistics include listings created after this one, the seller history includes this listing's own outcome, the vectoriser vocabulary includes words from future titles.
- The other half can be computed, but differently: the seller-history service lags a few hours, treats a missing seller as zero rather than null, and counts sales by settlement date rather than by order date. Every one of those is a small skew that the offline number never saw.
- The notebook features were written for a pandas frame; the serving features are written in a Java service by someone reading the notebook. The prose description of
seller_sales_30dis ambiguous about whether the window includes today.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict at listing creation whether a listing will sell within seven days. The label is a sale event within the window.
- The prediction is consumed by a promotion system that decides in the first minutes of a listing's life, so every feature must be computable at that moment.
- One example is one listing at creation: its attributes, its seller's history, category-level statistics, and text-derived features from the title.
- The notebook computed seller history over the entire event log, category statistics over all listings including future ones, and used a text vectoriser fitted on all titles.
- The serving system has the listing attributes at creation, a seller-history service with a few hours of lag, and no category statistics at all.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A feature is a function from a raw record and its context — history, related entities, aggregates — to a value, evaluated at a specific time. Its definition is the function; its value depends on the time of evaluation and the state of every source it reads. Two evaluations of the same definition at different times, or against differently-lagged sources, are different features (Train / Serve Skew).
- Some features carry fitted state: a scaler's mean, an encoder's vocabulary, an imputer's fill value, a target encoder's per-category rates. That state is learned from the training fold and must ship with the model; it is part of the artifact, not part of the serving code (Preprocessing Lives in the Artifact).
- Families of transformation recur across every tabular problem — aggregation over windows (Aggregation Features), bucketing and scaling (Bucketing & Normalisation), categorical encoding (Categorical Encoding, Target Encoding), temporal derivations (Temporal Features), missingness handling (Missing Data), interactions and domain-specific ratios. Each is a place where the two implementations can diverge.
- The features that dominate tabular models are the ones most exposed to the divergence: per-entity aggregates over time windows, which depend on a clock, a source and a null policy all at once.
A feature is a computation, not a column
The notebook's seller_sales_30d was one line of pandas. As a feature it is a window with a boundary, a source with a lag and a deduplication policy, a timestamp column to count by, a rule for sellers with no history, and an evaluation time. Every one of those was chosen implicitly by what the notebook happened to have loaded. The serving engineer chooses each one again, and there is no reason they agree.
Walking a single prediction through the system — the lab at /ml/prediction — makes this concrete: the listing is created, the request reaches feature retrieval, each feature is computed from whatever the serving sources hold at that instant, and the vector that reaches the model is the model's only view of the world. Every place that vector could differ from the training vector is a place the offline number did not measure.
Good ranking of sell-within-a-week on a held-out set, with forty notebook features computed over the full history.
Promotion decisions in the first minutes of a listing's life are close to random for new sellers and only modestly better for established ones; nothing errored.
- 1Half the features were leaky as defined — category statistics over future listings, seller history including this listing's outcome, a vocabulary fitted on future titles — and their honest as-of versions are much weaker.
- 2The rest were reimplemented from prose in a different language against lagged sources with a different null policy, so the serving vector differs from the training vector for the same listing.
- 3New sellers have no history at creation, which the notebook never had to face because it read the whole log; in serving they are the common case.
Fitted state belongs to the artifact
Some features carry state that was learned: the words in a vocabulary, the mean used to scale, the bucket edges, the per-category rates of a target encoder. That state was fitted on the training fold and it is as much a part of the model as the weights. If serving recomputes it — a vocabulary from recent titles, a mean from recent traffic — the model is applied to inputs from a different transformation.
The rule is that anything fitted ships with the model and is only applied, never refitted, in serving. It sounds obvious and is violated constantly, because the fitted state lives in a preprocessing step that is easy to think of as "cleaning" rather than as modelling (Preprocessing Leakage is the same mistake on the evaluation side).
Each feature is computed by one definition from equivalent sources in both paths, and any fitted state is loaded from the artifact rather than recomputed.
holds when Definitions are code shared or compiled to both paths; the artifact bundles the fitted preprocessing; a replay test compares vectors before promotion and on a schedule.
breaks when Either path is edited independently; a serving source changes lag or dedup policy; serving refits a scaler or vocabulary from traffic; a new category or null appears that the two null policies handle differently.
respond Find the diverging feature and fix the definition or the source on one side. Do not retrain until the paths agree; retraining on batch features rebuilds the same skew.
1// The definition is data: window, source, null policy, timestamp column.2// Both the batch job and the serving path are generated from it.3export const sellerSales30d = {4 name: 'seller_sales_30d',5 entity: 'seller_id',6 source: 'orders', // event table, not a rebuilt daily aggregate7 timestampCol: 'ordered_at', // not settled_at8 window: { from: '-30d', to: '0s', inclusiveEnd: false }, // [t-30d, t)9 agg: 'count',10 whenMissing: null, // unknown seller is null, never 011 asOf: 'request_ts', // training rows carry the same timestamp12} as constThe whenMissing and inclusiveEnd fields are the ones prose loses. A serving engineer defaulting an unknown seller to zero has created a category the model learned as "a seller with no sales", which is not the same as "a seller we have never seen".
The features that cannot be served
Category-level statistics computed over all listings, seller history that includes this listing's outcome, a vocabulary fitted on future titles — these are not skew, they are leakage, and they were strong in the notebook for that reason. The honest version of each is computed as of the listing's creation from data that existed then, and is weaker.
The remaining features fall into the families the rest of this module treats one at a time. Each family has its own way of diverging between training and serving, and its own fitted state to ship. The table is a map of where to look.
| Family | Example | Fitted state | Where training and serving diverge | Lesson |
|---|---|---|---|---|
| Aggregation | sales in last 30 days per seller | none | window boundary, source lag, dedup, null for unknown entity | Aggregation Features |
| Bucketing & scaling | price quantile, standardised age | bucket edges, mean and spread | refitted on serving traffic; edges shift with the population | Bucketing & Normalisation |
| Categorical encoding | one-hot category, ordinal condition | vocabulary, ordering | unseen category handling; vocabulary drift | Categorical Encoding |
| Target encoding | category default rate | per-category rates, smoothing prior | leaks in training unless out-of-fold; stale rates in serving | Target Encoding |
| Temporal | hour of day, days since last sale, lag-7 | none | timezone, as-of clock, window crossing the prediction time | Temporal Features |
| Missingness | indicator + imputed value | fill values | different null policies; missingness pattern changes upstream | Missing Data |
How to build it
Most important first.
- Define each feature once, as code that can run in both training and serving — a feature definition compiled to both a batch query and a serving computation — or compute it once in serving, log it, and train on the log (Feature Stores are one way to do this; not the only way).
- Give every feature an explicit evaluation timestamp and compute training features as of that timestamp from event history, so the training value is the value serving would have had (Point-in-Time Correctness).
- Write the null policy, the window boundary, the unit and the fitted state into the definition, not into prose. "Sales in the last 30 days" is not a definition;
[t - 30d, t)by order timestamp, null when the seller is unknown, is. - Test equivalence: replay a sample of production requests through the training feature path and assert the vectors match (Serving Contract Tests).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The mismatch rate between training-path and serving-path feature vectors on a replay sample, per feature, before promotion. This is the number that says whether the offline model and the deployed model share their inputs.
- Per-feature distribution distance between logged serving vectors and the training set on rollout day (Feature Drift).
- Offline metric improvements from a new feature are not evidence until the feature's serving computation exists and matches.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every feature the model consumes has one definition, executed from equivalent sources in training and serving, with a stated null policy and window boundary, and a replay test shows the two paths agree within tolerance.
- Every fitted state — vocabulary, scaler statistics, encoding tables — is stored in the artifact and applied unchanged in serving.
- The freshness of every source read at serving time matches the freshness assumed when the training features were computed as-of.
- Offline: for each feature, state its definition in code with the evaluation timestamp as a parameter; rebuild a sample of training rows with a stopped clock and diff.
- Offline: replay production requests through the training path and compare vectors feature by feature; block promotion on mismatches above tolerance.
- Online: log the serving feature vector, compare distributions to training on day one, and re-run the replay comparison on a schedule so a source change is caught.
What can go wrong
- The feature definition is shared, but the sources are not: training reads a warehouse table that is deduplicated nightly, serving reads a stream that is not, and the same definition yields different counts.
- A feature is dropped from serving because it is expensive, and the model is retrained without it — but the notebook features were selected together, and the remaining ones were never the best set on their own.
- The equivalence test passes at deploy, and six months later the seller-history service changes its lag from two hours to one day; nothing in the model pipeline notices.
- One definition for both paths forces one of them into the wrong tool — a warehouse query in a low-latency service or a service call in a batch job — and a translation layer between them is a new thing to maintain.
- Features that cannot be computed at serving time are simply lost, however predictive they were in the notebook, which lowers the honest number.
- Logging serving feature vectors for training is storage, a privacy question, and a delay: the first model on logged features cannot be trained until enough have been logged.
- "More features can only help; the model will ignore the useless ones." Every feature is a serving computation, a skew risk and a monitoring surface. A feature that helps a little offline and cannot be reproduced exactly hurts in production.
- "Engineering can reproduce the notebook from the description." A prose description is ambiguous about windows, nulls, units and sources; every ambiguity becomes skew. The definition has to be code.
- "Feature engineering is obsolete; a neural network learns its own features." For images, audio and text, largely true (Raw Features vs Learned Representations). For tabular data with entities, histories and windows, the aggregates still have to be computed by someone, and they still have to match at serving time.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a feature is a computation with a time, a source and a state — and must match at serving — holds for every model family; the specific transformations differ by modality.
- DATA-SPECIFICHand-engineered aggregates dominate on tabular data with entities and histories; on images, audio and free text, a pretrained network's learned representation replaces most of them, and the serving-equivalence problem shrinks to preprocessing such as resizing and tokenisation.
- CONTESTEDThe strongest case against a single shared feature definition is that it couples a batch warehouse and a low-latency service into one abstraction that serves neither well, and that a small team with a few models gets nearly the same protection from a replay-equivalence test at a fraction of the cost. That is right for small teams; the argument for the shared definition is that equivalence tests decay and do not cover timing.
Where the depth lives
This domain teaches the model and hands the rest off by name.