The Leakage Audit
A checklist run on every feature before the offline number is believed: when is it computed, from what, is it available at prediction time, does it correlate suspiciously, is it near-perfect on a subgroup.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The validation number is too good. What do you check, in what order, to find the leak before anyone commits to the number?
A lending team reports a default-risk model with near-perfect validation accuracy — dramatically better than the previous model and better than the credit bureau's score. The head of risk wants to ship it this week. A reviewer has one day to say whether the number is real.
Trust the number; it was produced by the standard evaluation code. If the reviewer cannot find a specific bug in a day, ship and monitor.
Near-perfect accuracy on a problem where the bureau, with decades of data, achieves far less is not a modelling breakthrough; it is a signature. The strongest feature turns out to be account_status, joined from servicing, which reads "charged off" for defaulted loans.
- Near-perfect accuracy on a problem where the bureau, with decades of data, achieves far less is not a modelling breakthrough; it is a signature. The strongest feature turns out to be
account_status, joined from servicing, which reads "charged off" for defaulted loans. - Removing it reveals a second leak:
collections_contactscounts calls made after the loan defaulted. A third:fraud_case_idis assigned only to loans referred after a missed payment. A fourth: applicants with several loans appear on both sides of the split, and repeat defaulters are identified by their attribute combination. - After all of these, the model is roughly as good as the previous one. The week of executive expectation was built on a column that says "this loan defaulted".
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict at application time whether a loan will default within twelve months. The label is the default event from the servicing system.
- The decision is approve, decline or refer, so the number that matters is ranking quality among applicants, at application time, with only application-time information.
- One example is one loan application with applicant, bureau and product features, joined to the loan servicing system, the collections system and the fraud team's case-management tool.
- The training table was assembled by an analyst under time pressure by joining current-state tables. The split was a random shuffle. Preprocessing was applied before the split. The validation number has been shown to the executive team.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Every leak route in this module leaves a specific trace that can be checked without waiting for production labels. Target leakage shows up as a feature written after the outcome and unavailable at serving time. Temporal leakage shows up as a window or a join that crosses the prediction timestamp. Entity leakage shows up as an overlap between the sides of the split. Preprocessing leakage shows up as a fitted step outside the fold. Evaluation leakage shows up as a test set with an unknown use count.
- The audit works because leaks are about *when*, and every feature can be asked when. For each feature: at what moment was this value computed, from what source, and would that source have held this value at the row's prediction timestamp? A feature that cannot answer is excluded until it can.
- The "too good to be true" heuristic is a prior. Near-perfect validation on a problem known to be hard is more likely to be a leak, a broken split, a duplicated entity, a target-derived column or a data bug than a breakthrough. The audit turns the prior into a procedure.
The audit, cheapest step first
The order matters. Reading the feature list and asking the source owners when each column is written costs an hour and finds most target leaks. The split assertion costs a line. The serving-availability check needs a dry run of the feature path. The stopped-clock rebuild and the later-period holdout are the expensive steps and are run last, on whatever survived.
Each step has a failure signature and a specific response, and the response is never "retrain on the same table". It is fix the feature, fix the split, or fix the order, and then measure again.
- 1When is it computed, from what?
For each feature, name the source table and the process that writes it, and state whether that process runs before or after the row's prediction timestamp.
fails by A column written by the outcome process or joined from a current-state table that is overwritten as the entity's life proceeds — target leakage.
- 2Is it available at prediction time?
Dry-run the serving feature path on a sample of live requests and compare per-feature non-null rates to the training table.
fails by A feature strong in training and null or default in serving — it was never known at prediction time.
- 3Does any window cross the snapshot?
For each windowed or lagged feature, assert the window end is at or before the prediction timestamp in event time; rebuild a sample with a stopped clock and diff.
fails by Windows anchored to job time, joins to rebuilt aggregates, load timestamps used as event timestamps — temporal leakage.
- 4Does any entity straddle the split?
Assert zero overlap of the entity key between train and validation; check high-cardinality features for near-unique values; deduplicate.
fails by Shared entities or near-duplicates across the split, or a column that identifies entities — entity leakage.
- 5Was anything fitted before the split?
Read the pipeline for imputers, scalers, encoders, selectors and rebalancing steps; confirm each is fitted inside the fold and shipped in the artifact.
fails by Selection or encoding on the full dataset; a hard-coded column list from an exploration — preprocessing leakage.
- 6Does it correlate suspiciously?
Rank features by label correlation and by importance; ablate the top one and retrain; run the shuffled-label test on the whole pipeline.
fails by A single feature the model collapses without, or a pipeline that scores above chance on permuted labels.
- 7Is it near-perfect on a subgroup?
Evaluate per slice — source system, branch, region, cohort — and look for a slice where the model is implausibly good.
fails by One source records a post-outcome field the others do not; the leak is confined to that slice and diluted in the average (Evaluation Slices).
- 8How many times has the test set been used?
Check the recorded evaluation count and the number of candidates compared; score the survivor on a fresh, later-period, unseen-entity holdout.
fails by A holdout fitted by a year of selection — evaluation leakage.
The last step is the one that catches what the others missed. If the audited validation number and the fresh-holdout number still disagree by more than their intervals, there is a leak the checklist did not name, and the number is not ready.
Near-perfect validation accuracy on a random split of the joined table, dramatically ahead of the bureau score and the previous model.
At application time every servicing, collections and case-management column is null; the model's strongest features are constants, its approvals are close to random among applicants, and the defaults that reveal it arrive over twelve months.
- 1
account_status,collections_contactsandfraud_case_idare all written after origination by the outcome process — target leakage joined from current-state tables. - 2Applicants with several loans straddled the random split and repeat defaulters were identified by attribute combination — entity leakage.
- 3Preprocessing was fitted before the split, a smaller contamination that on its own would not explain the number but was present.
Too good to be true
Near-perfect validation on a hard problem is not good news. The set of explanations — a target-derived column, a broken split, duplicated entities, a window into the future, a preprocessing step outside the fold, a data bug that copied the label into a feature — is far more probable than a breakthrough, and every one of them is checkable in a day. The heuristic is a prior, and the audit is what the prior demands.
The heuristic needs calibration to the domain. Some problems are genuinely easy and a clean, well-separated dataset gives a near-perfect honest number. The question is always: what does the best-resourced existing approach achieve on this problem, and is the gap between it and this number plausible from the features that were added?
looks like A categorical column from the loan servicing system with values such as current, late, paid_off, charged_off — an ordinary-looking status field with high feature importance.
why it leaks The servicing system overwrites the status as the loan proceeds. Joined today, it holds the end of each loan's story; charged_off is the default label under another name.
new or null at decision time. The feature is a constant, and the model has learned little else because it never needed to.fix Exclude every servicing, collections and case-management column written after origination; rebuild the remaining features as-of the application timestamp from history tables; re-audit.
| Option | Quality | Cost | Operational | Note |
|---|---|---|---|---|
| Read the feature list with the source owners | An hour; finds most target leaks and current-state joins; misses windows, entity overlap and selection. | |||
| Automated checks in the training job | Split overlap, stopped-clock rebuild, serving availability, shuffled labels, top-feature ablation; catches regressions every run; needs a serving dry-run path. | |||
| Fresh later-period unseen-entity holdout | The only check that catches an unnamed leak; needs history tables, as-of feature rebuild and labels that may take months. | |||
| Ship and monitor | Cheapest today; on a twelve-month label the leak is found a year of decisions later. |
caveat The scores cannot say that the options are cumulative rather than alternatives, nor that the value of the expensive holdout depends entirely on the label delay: with labels in a day, shipping to shadow is nearly as informative and far cheaper; with labels in a year, it is the only honest check available before the damage is done.
The audit is a gate, not a postmortem
The reviewer had a day because the number had already been shown to executives. The cheaper arrangement is that the audit runs on every training job, that its automated checks block promotion, and that its manual steps are recorded in a manifest shipped with the artifact. Then the number that reaches the executive team is the audited one, and there is no week of expectation to unwind.
This is also where the module hands over to the rest of the domain. The audited model is honest offline; whether it stays honest is a question of point-in-time correctness in the feature pipeline (Point-in-Time Correctness), of skew between training and serving (Train / Serve Skew), and of monitoring that would notice if a later pipeline change reintroduced a leak.
Every feature's audit entry — computed when, from what, available at serving, fitted inside the fold — remains true for the pipeline that is currently producing training and serving features.
holds when The automated checks run on every training job and block promotion; the manifest is versioned with the feature pipeline and re-verified on any change to a join, a window or a source.
breaks when A join is switched to a current-state table; a source changes its timestamp semantics; a new feature is added without an entry; a preprocessing step is added outside the pipeline object.
respond Treat an unexplained offline improvement as a leak until the audit says otherwise. Re-run the audit, fix the pipeline, and re-report.
1feature: prior_defaults_24m2computed_from: loan_events # event table with event_ts, not current state3computed_when: as_of application_ts # window [application_ts - 24m, application_ts)4writer: servicing default event # runs after outcome — but for PRIOR loans only5available_at_serving: true # dry-run non-null rate matches training6fitted_step: none7label_correlation: strong, plausible # bureau reports similar strength8subgroup_check: uniform across branches9audited_by: risk-review10audited_on: 2026-08-2011pipeline_version: features-v41The pipeline_version line is what makes the manifest checkable: when the feature pipeline version changes and the manifest does not, the audit is stale by construction and the promotion gate can say so.
How to build it
Most important first.
- Run the audit as a gate before the number leaves the team, not as an investigation after production disagrees. The order below is cheapest-first: the features are read before anything is retrained.
- For each feature, record the answers in a feature manifest — computed when, from what, available at serving, correlation with label, subgroup behaviour — and ship the manifest with the model so the next reviewer does not start from nothing.
- Automate what can be automated: the stopped-clock rebuild, the split overlap assertion, the serving-availability comparison, the shuffled-label test, the ablation of the top feature. These are data tests (Data & Feature Tests) and belong in the training pipeline.
- Treat the honest number as the result. A model that beats the previous one modestly after the audit is a success; a model that beats it dramatically before the audit is a bug report.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The validation number *after* every audit step, against a holdout of later, unseen applicants with features rebuilt as-of. The drop from the original number is the size of the leak; the residual is the model.
- The per-feature audit results themselves: how many features were excluded and why. A model whose ten strongest features all failed the audit had no model underneath.
- The original number is not evidence and should not be reported alongside the honest one as if they were alternatives.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every feature in the shipped model has a manifest entry stating when it is computed, from what source, and that it is available at serving time, and the entry has been re-verified since the last pipeline change.
- The split, the preprocessing order and the test-set use count have not changed since the audit that produced the reported number.
- The audit's automated checks — stopped-clock rebuild, split overlap, serving availability, shuffled labels, top-feature ablation — run on every training job and block promotion when they fail.
- Offline: run the pipeline device below in order on every candidate; record each step's finding in the manifest.
- Offline: re-score the audited model on a later-period holdout of applicants absent from training; compare to the audited validation number, and treat a residual gap as an unfound leak.
- Online: on the first day, compare per-feature serving availability and distribution to training; once labels arrive, compare production ranking quality to the audited number, not to the original.
What can go wrong
- The audit is run once, passes, and the feature pipeline later changes a join from a history table to a current-state table for performance; the manifest is now wrong and nobody re-runs the audit.
- The audit finds and removes the obvious leaks, the number drops to merely very good, and the review stops — with a subtler temporal window still present because "the big leaks are gone".
- The subgroup check is skipped because it is manual, and the model is near-perfect on one branch's loans because that branch's system records a post-default field the others do not.
- A thorough audit takes days per model and needs access to source-system semantics that the modelling team may not have; it is a coordination cost with data owners.
- A feature manifest is documentation that decays, and keeping it current across pipeline changes is ongoing work that no metric rewards.
- The audit systematically lowers reported numbers, which is the right outcome and an uncomfortable one when a higher number has already been promised.
- "We could not find a leak in a day, so the number is probably real." Absence of a found leak is not evidence of an honest number when the prior is this strong. The honest response is to build the later-period, unseen-entity, as-of holdout and score on it; if that cannot be done in a day, the number waits.
- "The model is near-perfect because the bureau features are very strong." The bureau's own score is not near-perfect on the same problem. A feature set that dramatically beats the best-resourced model in the industry is a leak until proven otherwise.
- "Ship it and monitor; production will tell us." Production labels take twelve months on this problem. A leaked lending model approves the wrong applicants for a year before the outcome is measurable (Ground-Truth Delay).
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALEvery feature can be asked when it was computed and from what, on any task or modality; the audit's steps are the same for a tabular lending model and an image classifier with session-level duplicates.
- DOMAIN-SPECIFICWhat "too good to be true" means depends on the domain's known difficulty: near-perfect default prediction is implausible because the bureau cannot do it, whereas near-perfect classification of a clean, well-separated industrial defect may be entirely real. The heuristic needs a domain prior, not a fixed threshold.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the audit's automated checks are a test suite over the training pipeline, and the promotion gate they feed is a release-engineering control; both live in a domain this platform links by prose rather than by id.