TestingGENERALDATA-SPECIFICCONTESTED

Data & Feature Tests

Schema, nulls, ranges, cardinality, target prevalence and distribution against the training reference — and the one test that catches most leakage: no feature timestamp may exceed its prediction time.

Target & data What to measure What must stay true ▶ ML Failure Simulator

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

What should be asserted about the training and serving data on every refresh, and which single assertion catches the failures that make offline metrics lie?

The problem

A credit team retrains monthly from a warehouse table their data platform maintains. The last retrain's validation metric jumped to a level the team lead calls "too good". Nobody changed the model. The table gained a column last month.

The obvious approach

The platform team owns data quality and runs their checks. The model team's test is the validation metric: if it is in the expected range, the data was fine.

Why it breaks

The validation metric was not in the expected range — it was above it — and that was the only signal. The new column is computed from account state months after origination, so for defaulted loans it encodes the default. The metric is excellent because the feature is the label (Target Leakage).

How it breaks — usually after the offline metric looked fine

The validation metric was not in the expected range — it was above it — and that was the only signal. The new column is computed from account state months after origination, so for defaulted loans it encodes the default. The metric is excellent because the feature is the label (Target Leakage).
In production, at application time, days_since_last_missed_payment is null for every applicant because there are no payments yet. The model learned that null means "never missed", the safest value. Every new applicant looks safe.
The platform team's quality checks passed: the column has the right type, a sensible null rate for the table as a whole, and values in a plausible range. Data quality is not the same as suitability for a model trained at a point in time.
A null-rate test on the serving features would have fired on day one — the column is null on every request — but the only serving test was a schema check, and the column was present.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

The surrounding model predicts default within twelve months of loan origination (Designing a Churn Prediction System has the same shape). This lesson's target is the training table and the serving features: are they what the model was designed for, on this refresh.
The label is a default event that becomes known months after origination, so every feature must be computable as of origination, not as of the row's last update.

Data

One row per loan: applicant features at origination, a set of account-history aggregates, and the twelve-month default label. The table is rebuilt nightly from the operational database.
The new column is days_since_last_missed_payment, computed by the platform team from the *current* account state, and joined by loan id.
Serving features are computed at application time from the same operational database, before any payment exists.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A data test is an assertion about a dataset, run when the dataset changes. The assertions that matter for a model come in two groups. Contract tests are the ones any consumer would write: id NOT NULL, age >= 0, the schema matches, cardinality of a categorical is within bounds, volume is within an expected range of the previous refresh. Reference tests compare the refresh to the dataset the current model was trained on: null rate per column, distribution distance per feature, target prevalence.
The test that distinguishes ML data testing from data quality is the point-in-time test: for every row, every feature's effective timestamp must be less than or equal to the row's prediction time. A feature computed from state after the prediction time is information from the future, and its presence is what makes an offline metric lie (Point-in-Time Correctness, Temporal Leakage). Running it requires that every feature carry, or be derivable to, an as_of timestamp — which is itself a data contract worth asserting.
Target prevalence is its own assertion. A label rate that moves between refreshes is either the world changing or the label construction changing, and the model cannot tell which; the test makes a human look (Label Construction).
Serving features need the same tests on a sample of live requests: null rate, range, cardinality, and distribution against the training reference. A feature that is always null at serving time and rarely null in training is the most common form of skew, and it is a one-line assertion.

The one inequality

For every row in the training table and every feature in that row, the feature's effective timestamp must be at or before the row's prediction time. A feature that violates it was computed from a state the model will not have at serving time, and if that state correlates with the label — as account state months after origination correlates with default — the model will learn the correlation and the validation metric will reward it.

The test is one join and one comparison. The precondition is that every feature carries an as_of timestamp reflecting the event that produced it, which is the part most pipelines lack. Adding it is the largest cost of ML data testing and the most valuable change a feature pipeline can make (Feature Pipelines on the data side).

leakagedays_since_last_missed_paymentThe column that arrived last month

looks like A well-typed integer column with a sensible null rate across the table, joined by loan id, that improved the validation metric sharply.

why it leaks It is computed from the account's current state, months after origination. For loans that defaulted it encodes the missed payments that constitute the default; the label reaches the model through the feature.

offline

The validation metric jumps well above the previous model's, because validation rows have the same post-origination state as training rows.

production

At application time there is no payment history; the feature is null for every applicant, which training taught the model to read as "never missed". Every applicant scores as safe.

fix The point-in-time test rejects the column; if the concept is useful, compute it as of origination from *prior* accounts, with an as_of that proves it.

when this feature is fine The same feature, computed from the applicant's earlier loans and stamped as of the origination date of the new one, is legitimately known at prediction time and may well be predictive.

The point-in-time test, as a query that must return zero rows

1-- every feature value must have existed at or before the prediction moment
2SELECT t.loan_id, f.feature_name, f.as_of, t.prediction_time
3FROM   training_rows t
4JOIN   feature_values f
5  ON   f.entity_id = t.applicant_id
6 AND   f.feature_name IN (SELECT feature_name FROM model_feature_list)
7WHERE  f.as_of > t.prediction_time         -- the violation
8   OR  f.as_of IS NULL;                    -- no timestamp: cannot be trusted
9 
10-- the second assertion: does the label rate look like last month's?
11SELECT refresh_date,
12       AVG(CASE WHEN defaulted THEN 1 ELSE 0 END) AS prevalence,
13       COUNT(*)                                     AS rows
14FROM   training_rows
15GROUP  BY refresh_date
16ORDER  BY refresh_date DESC LIMIT 6;

The IS NULL branch matters as much as the inequality. A feature without an as_of cannot be proven safe, and treating "unknown" as "fine" is how the new column got in.

Contract and reference: the rest of the checklist

The contract tests are the ones every consumer of the table should want: keys not null, ages not negative, types stable, categorical cardinality bounded, row count within range of the last refresh. They catch pipeline breakage. They are necessary and they are not ML-specific.

The reference tests are ML-specific because the reference is the model's training set. Each refresh is compared, feature by feature, to the distribution the current artifact was trained on — null rate, a distance on the histogram, prevalence of the label. A refresh that differs by more than the historical refresh-to-refresh variation is either the world moving or the pipeline changing, and the test's job is to make a person decide which before the model is retrained on it.

Assertion	Kind	Runs on	Catches
`id NOT NULL`, `age >= 0`, types match	contract	every refresh	pipeline breakage, bad joins
row count within range of previous refresh	contract	every refresh	a truncated window, a doubled load
categorical cardinality within bounds	contract	every refresh + serving sample	an id column mistaken for a category; a new upstream code set
null rate per feature vs training reference	reference	every refresh + serving sample	a feature that went null; serving-time skew
distribution distance per feature vs reference	reference	every refresh + serving sample	a unit change, a column meaning change, drift
target prevalence vs history	reference	every refresh	a label construction change; a real shift
`feature_as_of <= prediction_time`, no null `as_of`	point-in-time	every refresh	leakage — the future in the training set

The serving side of the same tests

The reference test that compares a refresh to the training set is the same test that compares a sample of live serving features to it. It runs on the other side of the deploy, and it is the earliest signal a model can give: a feature that is null on every request, a categorical with values the training set never had, a distribution that moved on rollout day — each shows before the first outcome label exists.

Run it on the serving path, not on a store fed from the warehouse. A sample drawn from the training path tests the training path a second time.

must stay trueServing features look like training features

For each feature, the live serving distribution and null rate match the training reference within the tolerance the historical variation justifies.

holds when The serving sample is drawn from actual requests; the reference is stored with the artifact and not regenerated; the null policy is the same on both sides.

breaks when A feature is unavailable at request time and defaults to null or zero; an upstream code set changes; a deploy changes the feature service; the world drifts.

how you would know A daily reference test on the serving sample, per feature, with null rate and distance; a page on a null-rate jump, since that is skew rather than drift (Feature Drift separates the two).

respond For a null-rate jump, fix the serving path. For a distribution shift with no pipeline change, decide whether it is drift worth retraining on — after the data tests pass on the refresh.

How to build it

Most important first.

Write the contract tests as a schema with expectations — type, nullability, range, cardinality bounds, volume bounds — and run it on every refresh, blocking training on failure (Data Contracts on the data side is the general form).
Require an as_of timestamp on every feature and run the point-in-time test on every refresh: feature_as_of <= prediction_time for every row. Fail the refresh on a single violation; a single leaked feature is enough.
Store the training reference — per-feature distributions, null rates, prevalence — with the artifact, and test each refresh against it with a per-feature distance and a tolerance set from historical refresh-to-refresh variation.
Run the same reference test on a daily sample of serving features. The serving null rate on a feature the model learned to rely on is the first thing to check after any deploy (Feature Drift).
When a column is added upstream, treat it as untrusted until it passes the point-in-time test; a new column that improves the metric is a leakage suspect before it is a win.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Point-in-time violations per refresh, which must be zero. Any nonzero count means the training table contains the future.
Per-feature distribution distance from the training reference, on the refresh and on the serving sample, with the historical variation as the baseline for the tolerance.
Target prevalence per refresh against its history.
The validation metric is not a data test. A metric that improves on a refresh with no model change is a signal to run the data tests, not a result.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Every feature in the training table has an effective timestamp at or before its row's prediction time, and the timestamp reflects the event that produced the value rather than the table build.
The per-feature distributions, null rates and target prevalence on each refresh stay within tolerances derived from historical variation of the training reference.
The serving features for a live request are drawn from the same distributions, with the same null policy, as the training features the artifact was built on.

How to verify — offline, online, and over time

Offline: inject a deliberately leaked column — a feature computed from post-prediction state — into a refresh and confirm the point-in-time test fails it and training is blocked.
On each refresh: the contract, point-in-time and reference tests run and their results are recorded with the resulting artifact.
Daily in production: the serving sample passes the same reference test, and a null-rate change on any feature pages before any outcome label exists.

What can go wrong

Failure modes in production

The as_of timestamp is stamped at table build time rather than at the event that produced the feature, so the point-in-time test passes trivially and catches nothing.
The reference distribution is regenerated from each refresh, so the test compares the refresh to itself and drift is never detected.
The tolerance is set so tight that every refresh fails and the test is disabled; or so loose that a column that went entirely null passes because its distance is "within range" of an old outage.
The serving sample is taken from a cached feature store that is populated from the warehouse, so it tests the training path twice and the serving path never.

What the recommended approach costs

An as_of timestamp per feature is a change to the feature pipeline and the storage format, and to every upstream producer who now has to supply one.
Reference tests need a tolerance, and the tolerance is a judgement that will be wrong in one direction or the other for some feature.
Blocking training on a data test failure means a monthly retrain can be late; not blocking means the failure ships.

Misreads

"The platform team runs data quality checks, so the data is tested." Data quality tests suitability for any consumer. A model needs the point-in-time test and the comparison to its own training reference, and no general-purpose quality check includes either.
"The metric went up, so the new column is a good feature." A metric that jumps on a column addition with no model change is the signature of leakage. The column is a suspect until the point-in-time test clears it.
"The column is present in the serving request, so the serving schema matches." Present and null is a schema match and a distribution mismatch. The null-rate test is the one that catches it.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThe point-in-time inequality and the reference comparison apply to every supervised model whose features are computed from state that evolves after the prediction moment, whatever the model family or the modality.
DATA-SPECIFICOn a static dataset with no temporal dimension — a one-off survey, a fixed image corpus — the point-in-time test is vacuous and the contract and reference tests carry the whole load; the inequality earns its place on any dataset built from operational records that update over time.
CONTESTEDWhether to block training on a reference-distribution test is disputed. The strongest case against blocking is that distributions legitimately shift with the world, that a blocked retrain leaves a stale model in production, and that the right response to drift is often to retrain sooner rather than to stop; the strongest case for blocking is that a distribution jump on a refresh with no known cause is far more often a pipeline bug than the world, and a human should look before the model learns it.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringdata-tests data-quality data-contracts contract-enforcement distribution-tests nullability-and-defaults volume-anomalies feature-pipelines