FeaturesGENERALDOMAIN-SPECIFICSIMPLIFIED

Missing Data

Why a value is missing is information. Imputation is fitted on the training fold and shipped, the missingness indicator is often the better feature, and the null policy must be identical at serving.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

A null is not a value. What does its absence mean, how should the model see it, and what has to match between training and serving for the answer to hold?

The problem

A telemedicine service predicts which incoming patient messages need a clinician within the hour. Several intake fields — temperature, pain score, medication list — are optional. The model validated well. After the intake form was redesigned to make fewer fields required, urgent messages started being triaged as routine.

The obvious approach

Fill missing values with the median so the model has complete rows. Missing means unknown, and the median is the least-wrong guess for an unknown.

Why it breaks

The pain score's missingness was itself the signal: patients too unwell to finish the form were the urgent ones. Imputing the median erased that signal and told the model those patients had average pain.

How it breaks — usually after the offline metric looked fine

The pain score's missingness was itself the signal: patients too unwell to finish the form were the urgent ones. Imputing the median erased that signal and told the model those patients had average pain.
When the form redesign made fields optional, the missingness rate tripled and its meaning changed — now a null pain score mostly means the patient skipped an optional field. The model had partly learned the old meaning from the values that remained, and it was now applied to a population where the same null meant something else.
The serving path imputed with a different median — computed from the last week of traffic — so even the fill value differed from training, and the drift in missingness was hidden by the fact that every row arrived complete at the model.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict whether a message will be escalated to a clinician within an hour. The label is the escalation event.
A false negative is a delayed urgent case; a false positive is clinician time. The costs are asymmetric and the model is thresholded accordingly.

Data

One example is one message with structured intake fields, many of them null, plus text-derived features.
Nulls were imputed with the training-set column median before modelling. No indicator of missingness was kept.
In the training period, a null temperature mostly meant the patient did not own a thermometer; a null pain score mostly meant the patient was too unwell to complete the form, which correlated with urgency.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Missingness has a mechanism. Missing completely at random (MCAR): the null is unrelated to anything — a dropped packet. Missing at random (MAR): the null depends on other observed features — older patients skip the app's pain slider. Missing not at random (MNAR): the null depends on the unobserved value itself or on the label — the sickest patients cannot finish the form. Only under MCAR does imputation lose nothing; under MAR the other features carry the information; under MNAR the null *is* the information and no fill value recovers it (What Is One Example? decides what a null means).
An imputer is a fitted transform: a median, a mean, a model that predicts the missing value from the others. It is fitted on the training fold, shipped in the artifact, and applied unchanged in serving (Preprocessing Leakage, Preprocessing Lives in the Artifact). Refitting it on serving traffic silently changes the input.
A missingness indicator — a binary is_null column per feature — gives the model the mechanism directly. For MNAR features it is frequently the strongest feature available. Tree models can also consume nulls natively by learning a default direction at each split, which is an indicator in disguise and just as sensitive to the null policy changing.
The null policy is a contract with the serving path and with upstream: what counts as missing (null, empty string, sentinel -1, 0?), what is filled with what, and which features carry indicators. A form redesign, a schema change or a new client version changes the missingness pattern without anyone changing the model (Feature Drift on the indicator).

Why is it missing?

The null pain score was not an unknown pain score. It was a patient who could not finish the form, and that was the strongest urgency signal in the intake. Imputing the median told the model that patient reported average pain. The model was then asked to detect urgency with its best clue replaced by a bland value, and validation — computed on rows that had been imputed the same way — could not see what had been removed.

The three mechanisms are three questions to ask of every nullable feature. Does the null depend on nothing (a dropped packet)? On other observed fields (older patients skip the slider)? On the unobserved value or the label itself (the sickest cannot finish)? The last is where the null is the feature, and where imputation without an indicator destroys it.

leakagepain_score (null for patients too unwell to finish)Not leakage — the signal imputation erases

looks like A numeric field with a moderate null rate, imputed with the median so the model sees complete rows.

why it leaks It does not leak; the opposite. The null is legitimate, prediction-time information about the patient's state, and median imputation replaces it with a value asserting the patient is typical. The model loses its best clue on exactly the urgent cases.

offline

Validation is computed on imputed rows, so the loss is invisible; the model looks fine because it never had the signal to lose in the evaluation either.

production

Urgent patients who cannot finish the form arrive with a median pain score and are triaged as routine, and the false-negative rate on the sickest patients is far above what validation suggested.

fix Keep pain_score_missing as a feature; impute the value only so the model has a number; fit the fill on training and ship it; monitor the missingness rate.

when this feature is fine The null is a fine feature precisely because it is known at prediction time: the form has been submitted, the field is empty, and the serving path sees the same emptiness training saw — as long as the null policy that turns that emptiness into an indicator is the same in both.

Mechanism	The null depends on	Example	Indicator carries	Imputation
MCAR	nothing	a sensor packet dropped	nothing	any reasonable fill; interpolation for series
MAR	other observed features	older patients skip the app's pain slider	little beyond what age already carries	fill conditional on the observed features; indicator cheap to keep
MNAR	the missing value itself, or the label	the sickest patients cannot finish the form	a great deal — often the strongest feature	no fill recovers it; keep the indicator and fill only so the model has a number

The fill is fitted; the policy is shared

Whatever fills the null — a median, a model — was fitted on the training fold and belongs in the artifact, like a scaler or a vocabulary. The serving path that computed its own median from last week's traffic applied a different transformation with the same name. And the policy for what counts as missing is upstream of the fill: a null, an empty string, a 0 from a client that defaults numeric fields, a -1 sentinel — each must be recognised identically or the indicator is silently never set.

Both paths should run the same code for this. It is small code, and the cost of two implementations is a null that means "missing" in training and "zero" in serving.

Null policy and imputer, fitted on train and shared

1MISSING_SENTINELS = {None, "", "null", -1}     # the contract with upstream
2 
3def is_missing(v):
4    return v in MISSING_SENTINELS or (isinstance(v, float) and np.isnan(v))
5 
6def fit_imputer(train_df, cols):
7    return {c: train_df[c][~train_df[c].map(is_missing)].median() for c in cols}
8 
9def apply(row, fills):
10    out = {}
11    for c, fill in fills.items():
12        missing = is_missing(row.get(c))
13        out[c + "_missing"] = int(missing)                # the indicator: the signal
14        out[c] = fill if missing else row[c]              # the fill: just a number
15    return out
16 
17artifact["fills"] = fit_imputer(train, ["temperature", "pain_score"])
18# serving imports is_missing, apply and artifact["fills"]; it fits nothing

The MISSING_SENTINELS set is where upstream changes land. A new client that sends "N/A" is not in it, and its nulls become string-parsing errors or zeros rather than indicators — which the missingness-rate monitor shows as a *drop* in nulls, not a rise.

When the meaning of a null changes

The form redesign did not break the pipeline. It changed why a pain score was missing: from "too unwell to finish" to "skipped an optional field". The indicator's rate tripled and its relationship to urgency collapsed. The model kept using it with its old meaning, and the imputed rows kept looking complete, so nothing in the serving path objected.

This is concept drift on a single feature (Concept Drift), and the missingness-rate monitor is the early signal: a step change in the null rate for one field, on the day of a product release, is an upstream change to what the null means. The response is a decision — can the old signal be recovered by asking the question again, or does the model need to learn the new meaning — and it needs a person.

Triage on the null-pain-score subgroup, illustrative counts

Predicted Escalated to a clinician within the hour

Predicted Handled as routine

Actual Escalated to a clinician within the hour

True positive

caught Escalated to a clinician within the hour

False negative

missed Escalated to a clinician within the hour

Actual Handled as routine

False positive

Handled as routine flagged as Escalated to a clinician within the hour

True negative

151

correctly left alone

n = 200precision = 0.667recall = 0.450accuracy = 0.845

a false positive costs A clinician reads a routine message: minutes of clinical time, recoverable.

a false negative costs An urgent patient waits in the routine queue for hours: the harm the service exists to prevent, and it lands on the patients who could not finish the form.

The overall accuracy on this subgroup looks respectable because routine messages dominate; the false-negative count is the row that matters, and it is what the erased missingness signal cost. Counts are for the shape of the argument.

must stay trueA null still means what it meant

The mechanism that produces a null in each indicator feature is the one that held in the training period, its rate is stable, and both paths recognise the same values as missing and apply the same fills.

holds when The intake form, client versions and upstream schemas are unchanged, or changes are mapped in the shared null policy; missingness rates match training; fills come from the artifact.

breaks when A form field becomes optional or required; a client sends a new sentinel; a schema change renames or defaults a field; the serving path refits the imputer; a tree implementation changes its null routing.

how you would know Missingness rate per feature at the serving boundary with a step-change alert; the label rate among null rows as labels arrive; a replay test of null handling; production recall stratified by indicator.

respond Identify the upstream change; decide whether the old signal is recoverable; if the meaning has genuinely changed, retrain with the new mechanism and re-validate recall on the null-row subgroup.

How to build it

Most important first.

Keep an indicator for every feature with meaningful missingness, and let the model use it; impute the value only so the model has a number, not to hide the null.
Fit the imputer on the training fold, ship it, and apply it unchanged; never compute a fill value from serving traffic.
Write the null policy as code shared by both paths: the set of sentinel values that mean missing, the fill per feature, the indicator per feature. Test that the serving path produces the same vector for a null as the training path did.
Monitor the missingness rate per feature at the serving boundary; a step change is an upstream change to the meaning of the null and needs a human decision, not a silent fill.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Missingness rate per feature at serving against training, daily. This is the number that would have caught the form redesign on the day it shipped.
Recall on urgent cases stratified by whether the row had nulls in the indicator features, once escalation labels arrive.
Validation on complete-looking rows measures a model whose inputs have already had their most informative signal filled in.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

The missingness mechanism for each indicator feature — why a null occurs — is the same in production as in the training period, and its rate is monitored.
The set of values treated as missing, the fill per feature and the indicator per feature are identical in training and serving, with fills loaded from the artifact.
No serving component fills or drops nulls before the feature path sees them.

How to verify — offline, online, and over time

Offline: for each feature, compare the label rate among null rows and non-null rows; a large difference is MNAR and the indicator must be kept.
Offline: replay serving requests with nulls through both paths and assert the vectors — fills and indicators — match.
Online: monitor missingness rates and the label rate among null rows as labels arrive; a rate change with an unchanged label rate is a benign upstream change, a changed label rate is a changed mechanism.

What can go wrong

Failure modes in production

Indicators are kept, but a new client version sends an empty string instead of a null, which the policy does not recognise as missing; the indicator is never set and the value is parsed as zero.
The missingness rate monitor fires on the redesign day, is attributed to the redesign, and is muted — while the model keeps using an indicator whose meaning has changed.
A tree model handles nulls natively and the team concludes there is no imputation state to ship; the serving library uses a different default direction for nulls.

What the recommended approach costs

Indicators double the feature count for sparse data and add columns that are exactly the ones most exposed to upstream form and schema changes.
Keeping the null policy in shared code couples the serving service to the training pipeline's definitions.
Treating a missingness-rate change as a human decision rather than an automatic retrain slows response and requires someone to own the question.

Misreads

"Impute, then the model has clean data." The model has data with its most informative signal removed, on the cases — the sickest patients — where the signal mattered most.
"The tree handles nulls natively, so there is nothing to ship." The default direction per split is learned state; the serving implementation must use the same one, and the null policy upstream must still match.
"Missingness went up after the redesign, so retrain on new data." Retraining teaches the new meaning of the null, which is probably right — but first decide whether the old signal is recoverable from a different field, and whether the form should ask the question again.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThat missingness has a mechanism and that the imputer is fitted state holds for any model family and any modality with structured inputs.
DOMAIN-SPECIFICIn clinical and self-reported data, MNAR is the norm — the null depends on the patient's state — and the indicator is often the strongest feature. In instrumented sensor data, nulls are usually MCAR dropouts and an indicator carries nothing; imputation by interpolation is the right tool there.
SIMPLIFIEDMCAR / MAR / MNAR is presented at the concept level as a way to ask why a null occurs; the formal definitions concern conditional independence of the missingness indicator, and the categories are not always distinguishable from the observed data alone. Any numeric contrast in this lesson is for the shape of the argument.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringnullability-and-defaults schema-evolution data-quality-dimensions semantic-changes