ContractsENGINE-SPECIFICSIMULATEDGENERAL

Breaking Schema Changes

A numeric field starts arriving as a string. Some consumers error; the dangerous ones cast, get null, keep every row, and report zero — with completeness, uniqueness and freshness all green.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

When a field's type changes upstream, which consumers fail loudly, which fail quietly, and what does the dashboard show while nobody is being paged?

Who needs this

Every consumer of the field, split into two populations that behave completely differently. Strict readers — a declared ingestion schema, a typed deserialiser, a warehouse column with a fixed type — get an error and are fixed within hours. Permissive readers — a SAFE_CAST, a TRY_CAST, an inference-based loader, a warehouse that widens a column to string — get a value and are fixed when a human notices the number (The Pipeline Succeeded. The Data Is Wrong.).

What one row is

The unit is one field, one record, one cast. The failure is decided per record at the moment of conversion: the engine either raises, which stops the batch, or returns null, which keeps it. Nothing about the row count, the partition or the job changes either way, which is exactly why the second outcome is invisible.

The obvious build

Cast defensively. Wrap the conversion so a malformed value cannot fail the run, because a job that dies on one bad record at 3 a.m. is a bad job. Every engine offers a permissive cast for exactly this reason and it is usually the default in transformation tooling (SQL Transformations).

Why it breaks

The permissive cast is applied to *every* record rather than to the rare bad one. A type change upstream makes all of them unparseable, and the defensive cast converts a total failure into a table full of nulls that publishes normally.

How it breaks with real data

The permissive cast is applied to *every* record rather than to the rare bad one. A type change upstream makes all of them unparseable, and the defensive cast converts a total failure into a table full of nulls that publishes normally.
Row counts match at every stage, so a completeness check comparing source rows to serving rows passes with nothing to report (Missing Rows).
Every order id is present exactly once, so a uniqueness check passes (Duplicate Rows).
The table is published on time, so a freshness check passes and the dashboard is up to date — showing zero (Freshness Checks).
SQL aggregates ignore nulls, so a sum over the emptied column returns null — and almost every model or BI layer wraps a monetary sum in a coalesce to zero, so what reaches the tile is a confident zero rather than a blank cell that would have prompted a question.
Had only some records failed to parse, it would be worse still: AVG() drops nulls from its denominator as well as its numerator, so the tile reports a perfectly plausible average of whichever records happened to survive (Nullability & Defaults).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

A type change is not one event but two: the producer emits a different physical representation, and *your* code decides what to do about it. The producer's half is usually announced badly; your half is a configuration choice you made months earlier and forgot (Schema Evolution).
Permissive casting exists because real data has rare bad records and a pipeline that dies on each one is unusable. It is the correct tool for a 0.01% malformed rate and precisely the wrong tool for a 100% one, and nothing in the cast distinguishes the two cases (Data Quality).
Null is the mechanism that makes this quiet. A raised exception stops the pipeline and creates an incident with an owner. A null keeps the row, keeps the count, keeps the schedule, and moves the failure from the orchestrator to a human reading a chart (The Pipeline Succeeded. The Data Is Wrong.).
Aggregate semantics finish the job. SQL aggregates skip nulls rather than propagating them, so a sum over an emptied column returns null and a coalesce — which almost every monetary model carries — turns that null into a zero. The wrong value therefore arrives at the dashboard in the most plausible possible form.
The mitigation is not a better cast. It is a check that refuses the batch — which converts a silent zero into a failed run, a stale table and a page (Contract Enforcement).

One change, two completely different populations of consumer

The producer ships a change that, from inside its own codebase, is reasonable: amount becomes a string so it can carry a scale and avoid a rounding argument. Downstream, that single change splits every consumer into two groups whose experiences have almost nothing in common.

The first group declares its types. An ingestion job with a fixed schema, a typed deserialiser, a warehouse column defined as an integer — all of them raise, the batch stops, and someone is paged. This group has a bad morning and correct data.

The second group coerces. It gets a value, it keeps the row, and it publishes on schedule. This group has a normal morning and wrong data, and it will keep having normal mornings until a person notices the number. Read the silent column below: the loud failures are the fortunate ones.

A numeric measure becomes a string

Before

order_id: string
placed_at: timestamp
amount: integer (minor units)
currency: string

After

order_id: string
placed_at: timestamp
amount: string ("1250" / "12.50")
currency: string

change The producer changes amount from an integer in minor units to a string. Their own service is unaffected: it parses its own field. The contract, if there was one, said integer.

Consumer	Effect	How it shows up
Ingestion job with a declared schema	Rejects the batch on a type mismatch. Nothing is written, the pipeline stops, and an engineer is looking at it within the hour.	Loudly — it raises
Typed deserialiser in a streaming consumer	Throws on the first record. The consumer stops advancing and lag starts climbing, which is a metric people already watch (Offsets and Commits).	Loudly — it raises
Transformation using a permissive cast	Every cast returns null. Every row survives. The measure column is entirely null and the run succeeds normally.	Silently — no error, wrong result
Revenue model summing that column	The sum skips every null and returns null; the coalesce the model carries for empty days turns that into a zero. The dashboard shows a confident zero for a normal trading day.	Silently — no error, wrong result
Average order value tile next to it	The numerator collapses while the order count does not, so a second metric moves alongside the first. Had only part of the batch failed to parse, `AVG` would have dropped those rows from its denominator too and reported a plausible average of the survivors.	Silently — no error, wrong result
Warehouse loader that widens the column to string	The column type changes in the target table. Old partitions hold integers, new ones hold strings, and every query spanning the boundary now depends on the engine (Forward Compatibility).	Silently — no error, wrong result

The cast that returns null instead of raising

ENGINE-SPECIFICThe permissive cast is spelled differently across engines and the exact syntax above will not run everywhere; some engines also expose the coercion count directly rather than requiring the two-pass form. What transfers is the pattern of counting coercions and thresholding per column, not this SQL.

The entire failure reduces to one line of SQL, and to a choice made long before the incident. Every major engine offers two casts: one that raises on unparseable input and one that yields null. Transformation tooling frequently generates the second, because a job that dies on a single malformed record is genuinely painful to operate.

The defensive cast is correct for a field where malformed values are rare and expected. It is catastrophic for a field where they suddenly become universal, and the cast has no way to tell those two situations apart — it sees one record at a time and there is nothing wrong with any individual decision it makes.

The fix is not to remove the permissive cast. It is to make "how many did we coerce" a number the pipeline knows and can fail on. A cast that silently produces null is a data-loss event with no counter; a cast paired with a null-rate assertion is a rare-record tolerance with a limit.

Two ways to handle a value that will not parse

Coerce and continue

Wrap every risky conversion in a permissive cast so the run cannot die. Nulls flow downstream, aggregates absorb them, and the pipeline's success rate looks excellent.

Coerce, count, and refuse above a threshold

Keep the permissive cast for its real purpose, but count coercions per column per run and fail the run when the rate exceeds what that column tolerates. Route the raw payload to quarantine so the original values survive the incident.

The permissive cast is a per-record decision and the problem is a per-batch property. No amount of care applied one record at a time can distinguish a rare malformed value from a systematic type change, because the two look identical at that granularity. Lifting the decision to the batch — where the rate is visible — is the only place the distinction exists.

The same transformation, with and without a limit on how much it will tolerate

1-- The failure, in one expression. On a normal day this coerces a handful of
2-- malformed records and saves the run. On the day the type changes upstream,
3-- it coerces every record and saves the run.
4select
5  order_id,
6  placed_at,
7  try_cast(amount as bigint) as amount_minor   -- null on failure, never raises
8from raw_orders;
9 
10-- Same tolerance, with a limit. The cast still protects the run from rare bad
11-- records; the assertion decides what "rare" means for this specific column.
12with cast_attempt as (
13  select
14    order_id,
15    placed_at,
16    amount                        as amount_raw,
17    try_cast(amount as bigint)    as amount_minor
18  from raw_orders
19),
20coercion_rate as (
21  select
22    count(*)                                                as rows_total,
23    count(*) filter (where amount_minor is null
24                       and amount_raw is not null)          as rows_coerced
25  from cast_attempt
26)
27select *
28from cast_attempt
29where (select rows_coerced::double precision / nullif(rows_total, 0)
30       from coercion_rate) < 0.001;   -- above this, publish nothing
31 
32-- Note what the last clause does: it does not clean the data, it refuses to
33-- publish it. That is the whole difference between a defensive pipeline and
34-- a pipeline that defends the number.

The counter is the part that matters. rows_coerced distinguishes "three bad records, as usual" from "every record, since 04:00", and those two facts are indistinguishable to the cast itself.

What the model does with this exact fault

SIMULATEDThese outcomes are produced by the in-repo pipeline model under a fixed seed, not measured on a real platform. The model deliberately holds everything else constant so that only the effect of this one fault is visible; a real incident would have noise around it that the model does not reproduce.

This platform ships a deterministic model of the pipeline, and this failure is one of its faults. It generates a fixed set of source orders, pushes them through source, change capture, broker, raw landing, transformation, warehouse and dashboard, and reports the number the dashboard shows against the number that is actually true.

Run with the schema-change fault and nothing else, the model reports the same row count — four thousand — at every single stage from source to serving table. Nothing is lost, nothing is duplicated, and the run publishes on schedule with the normal lag. Reported revenue is exactly zero; true revenue is not. Four of the model's six checks pass.

Turn on the contract check and the picture inverts. The batch is rejected, the run fails, nothing is published, and the serving table still holds the previous period. Now four checks fail instead of two — including freshness, which is the one most platforms actually page on. The validity check, ironically, now passes: there are no rows, so there are no invalid ones.

That inversion is the lesson. Enforcement did not reduce the number of failing checks; it increased it. What it changed is *which* checks fail, and it moved the failure from a category nobody monitors into a category everybody monitors (Contract Enforcement).

SIMULATED — the six checks, with and without the contract check

Check	Expresses	Catches	Still misses
Completeness — source rows versus serving rows	Every order the source recorded reached the serving table.	Nothing here without enforcement: it passes, because all four thousand rows arrived. With enforcement it fails, because nothing was published at all.	Exactly this failure. Row-level completeness is preserved by a cast that nulls values, which is why a count-based reconciliation is the wrong instrument for it.
Uniqueness — one row per order id	Each real order appears exactly once.	Nothing here, in either configuration. It passes both times.	Every value-level failure. Uniqueness is a statement about keys and says nothing about the columns beside them.
Freshness — newest complete record versus now	The table is recent enough for the decisions it drives.	Nothing without enforcement — the table publishes on time. With enforcement it fails immediately, because the run did not publish and the table still holds the previous period.	Data that is perfectly fresh and completely wrong, which is precisely the unenforced case here.
Validity — every amount non-null and numeric	The measure actually has values in it.	The failure, directly and immediately: the model reports every published row with a null amount after the cast.	A value that is well typed and wrong. It also passes vacuously under enforcement, because a table with no rows has no invalid ones — a check passing on an empty set is not evidence.
Distribution — today's shape against its own history	The day looks like the days before it, by volume and by category mix.	Nothing without enforcement: the country mix and total volume are untouched, because only the measure was destroyed. With enforcement it fails, because there are no rows to compare.	Any error that preserves the shape while changing every value inside it, which is this failure exactly.
Reconciliation — summed measure against the source	Revenue in the serving table equals revenue in the source for the same closed period.	The failure, in both configurations. This and validity are the only two checks that see it when enforcement is off.	Anything wrong identically at both ends, and any period that is not yet closed — which is where the newest and most-watched data lives.

SIMULATED, from src/de/sim/pipeline.ts under the schema-change fault. Two of six checks fail without the contract check; four of six fail with it. The count went up and the platform got safer, because the four include the one that pages.

Working backwards from the symptom

You will not meet this failure as a type error. You will meet it as a number, usually reported by someone who is not on your team, usually a day or more after it started. The table below is what the investigation actually looks like.

The single most useful move is to compare the raw payload before and after the boundary date. Raw is where the truth is — the original strings are there if nobody cast on ingest — and comparing two payloads settles in minutes what a conversation with the producing team can take a day to establish.

The second most useful move is to check whether the column was always this way. A column that is 0% null before a date and 100% null after it has a boundary, and that boundary is a deployment (Debugging a Data Incident).

From "revenue looks wrong" to a cause

Trigger	Symptom	Cause	Response
A revenue tile shows zero for a normal trading day.	Order counts are completely normal; the measure is zero.	The measure column is entirely null after a cast, and the sum ignores nulls.	Check the null rate on that column by day. If it steps to 100% on a specific date, this is a type or population change, not a business event.
Null rate on one column steps from 0% to 100% overnight.	Row counts, uniqueness and freshness all normal.	A permissive cast now failing on every record, or a producer that stopped populating the field.	Read the raw payload either side of the boundary. A string where an integer used to be identifies it immediately, and the distinction matters because the remedies differ.
A different consumer of the same field is paging with parse errors.	Two consumers of one dataset, one broken loudly and one apparently fine.	The same type change, met by a strict reader and a permissive one.	Treat the loud consumer as the diagnosis for the quiet one. When two consumers of a field disagree about whether it is healthy, the strict one is right.
A query spanning several months errors on a type mismatch.	Recent partitions and older partitions disagree about the column's physical type.	A warehouse loader widened the column, so history and new data are stored differently.	Read through a view that casts both sides explicitly, and schedule the history rewrite separately. Do not widen further in the middle of an incident (Forward Compatibility).
The boundary check fires and the run fails at 3 a.m.	The table is stale and the on-call engineer has an outage rather than a wrong number.	Enforcement working exactly as designed.	Route to the producing team, quarantine the batch rather than dropping it, and do not disable the check to restore the pipeline — a disabled check is how this failure becomes silent again (Contract Enforcement).
A finance close does not reconcile with the operational system.	The gap is discovered by someone outside the data team, weeks later.	This failure, undetected, for the length of the reporting period.	Reprocess from raw, restate the period, and tell every consumer who was given the earlier figure. Then add the validity threshold, because this is the check that would have made it a one-day incident (Reconciliation).

How to build it

Most important first.

Assert the type at the boundary rather than coercing it in the transformation. A conformance check that rejects a batch whose amount is not an integer catches this before any transformation has a chance to be defensive (Contract Enforcement).
Reserve permissive casts for fields where a malformed value is genuinely expected and rare, and pair every one of them with a null-rate threshold that fails the run when the rate crosses what "rare" means for that field (Data Tests).
Add a validity test on every measure that feeds a published metric: non-null rate, and a plausible range. It is the check that separates this failure from every other failure in the module (The Dimensions of Data Quality).
Reconcile a closed period against the source on a summed measure, not only on row count. Row count is exactly the check this failure is designed to pass (Reconciliation).
Alert on a metric going to zero as loudly as on a metric going missing. Most alerting treats zero as a value; for an additive business measure it is almost always an incident (Quality Alerting).
Keep raw immutable so the original strings survive the incident. If the cast happened on ingest, the values are gone and the only recovery is a re-extract from a source that may have moved on (The Raw Landing Zone).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A permissive cast guarantees the pipeline will not fail on this field. That is the entirety of what it promises, and it is a promise about the pipeline rather than about the data.
Row-level guarantees are all preserved by this failure: every source row is present, exactly once, on time. This is the clearest illustration in the domain that completeness and correctness are different properties (The Dimensions of Data Quality).
A contract check at the boundary guarantees that data violating the declared type does not enter. It explicitly does not guarantee that data will arrive at all — refusing bad data means having no data (Contract Enforcement).
Nothing guarantees that a zero is questioned. That depends on a human, on the alerting policy for that metric, and on whether the number went up or down.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check that catches this directly is a validity test on the measure: assert that the non-null rate of amount is above a threshold, and fail the run if it is not (Data Tests).
It misses a value that is well typed and wrong — an amount in the wrong currency, or in major units where the contract says minor, passes every type and null check there is (Semantic Changes).
A reconciliation of the summed measure against the source catches this too, and catches more besides, but only for closed periods and only where a source-side aggregate is actually available (Reconciliation).

Freshness

Untreated, this failure costs no freshness whatsoever. The table is published on schedule with the usual lag, which is why the freshness check is one of the four that stay green.
Treated with a boundary check, freshness is what you spend: the run fails, nothing is published, and the table holds the previous period until a human fixes the producer. That is the trade and it is the correct one (The Freshness SLO).
The recovery, once the producer is fixed, is a reprocess of the affected range from raw — which costs a second pass over the period rather than a permanent gap, provided raw kept the strings (Reprocessing vs Retrying).

When the schema or meaning changes

This *is* an evolution failure, so the recursive question is what to do next time. The answer is the expand-and-contract sequence: the producer adds amount_text alongside amount, consumers migrate, and the old field is removed once reads have gone to zero (Expand and Contract Migrations).
A type change applied to a table also splits history in two. Old partitions hold the old physical type, and a query spanning the boundary either errors or resolves to null on one side (Forward Compatibility).
The safest general rule is that a field's type is part of its identity. Changing it is a new field with a new name, not a modification of an existing one.

How to re-run this safely

Reprocess the affected range from raw once the producer is fixed and the boundary check is in place. Everything needed is still on disk if raw was landed unmodified (Keeping Raw History: The Recovery Position and the Liability).
Publish the corrected range atomically and tell the consumers who saw the zero. A restatement that nobody is told about is indistinguishable from a second bug (Atomic Publish).
If the cast happened on ingest and raw holds the already-nulled values, the strings are gone and recovery depends on the source still holding the range — which for a stream past its retention window it does not (Retention and Replay).

What can go wrong

Failure modes

The permissive cast, doing exactly what it was configured to do.
A boundary check that inspects only structure and not types, so a string in an integer field passes.
A validity test written as "not null" on a field that is legitimately nullable, so the threshold was set to zero and nothing fires.
Alerting that treats zero as a value rather than as an anomaly for an additive measure (Quality Alerting).
The mitigation's own failure: the boundary check rejects the batch, the run fails, and the on-call engineer disables the check to restore the pipeline — which is the worst possible outcome and a predictable one if the check has ever produced a false rejection (Contract Enforcement).
Raw that was cast on ingest, leaving no copy of the original values (The Raw Landing Zone).

Misreads

"Schema changes are safe as long as the column still exists." This lesson is the counter-example in its purest form: the column exists, every row is present, and every value in it is gone. Existence is not the property that matters (Semantic Changes).
"The row counts matched, so nothing was lost." Row counts are preserved by construction here. A count-based reconciliation is exactly the check this failure passes (Reconciliation).
"The job would have failed if something were wrong." The job did not fail because it was configured not to. That configuration is the failure (The Pipeline Succeeded. The Data Is Wrong.).
"Zero is a value, not an error." For an additive business measure over a normal trading day, a zero is an incident until proven otherwise, and treating it as data is how a whole day passes before anyone asks (Stale Dashboards).

Operating it

How you see it in production

Null rate per column per run, with a threshold per column rather than a global one. This is the highest-value chart in the module (The Data Quality Dashboard).
Summed measure per day against its own history — a measure that goes to zero while row counts hold is this failure's exact fingerprint (Volume Anomalies).
Counts of permissive-cast failures where the engine exposes them, which turns a silent coercion into a countable event.
The schema of the arriving payload, logged per batch, so "when did the type change" is answerable without asking the producer (CDC and Schema Drift).

What changes at 10x and 100x

At 10x volume nothing about this failure changes — it is a per-record type decision and it is equally invisible at any size.
What scales is the blast radius: a field consumed by fifty models produces fifty wrong tables from one cast, in dependency order, before anyone looks (Impact Analysis).
At 100x consumers, the probability that someone notices the zero quickly goes up, and the probability that someone has already made a decision on it goes up faster.

What drives cost here

The boundary check costs a type inspection per record per batch, which is proportional to records ingested and is small relative to any transformation over the same data (Scan Cost).
The reprocess after an incident costs a second pass over the affected range — proportional to the length of the range, which is proportional to how long the failure went unnoticed. Detection time is therefore a cost driver, not just a trust driver.
The uncosted item is the restatement: telling consumers that a published number was wrong, and the work they do in response, which is usually larger than the engineering fix.

What this approach costs

Strict boundary checking converts a silent wrong number into a loud outage. That is unambiguously the right trade and it costs availability of the data, and it will page someone at 3 a.m. for a change made by a team that is asleep (Contract Enforcement).
Permissive casting genuinely protects a pipeline from rare malformed records. Removing it entirely trades one real problem for another; the resolution is a threshold rather than a binary.
Validity thresholds have to be set per column, which is real, boring, ongoing work. A global threshold is either so loose it never fires or so tight it fires constantly (Alert Fatigue: The Page Nobody Reads).

Schema evolution lab

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

Schema evolution lab

The useful axis is not breaking against non-breaking. It is loud against silent — and a contract's whole job is moving changes from the second column into the first.

Change

BREAKS SILENTLYSend a number as a string

The cast produces null rather than an error, so the rows survive, the row counts match, and every amount silently becomes nothing.

With a contract: The batch is rejected because the payload no longer matches the agreed type. A silent wrong number becomes a loud outage — the correct trade, and it will still page someone at 3 a.m.

Change	No contract	Why
	compatible	Readers that do not know the column ignore it; readers that do get nulls for old rows, which is what they should get.
	breaks loudly	Old rows have no value for it, so anything that enforces the requirement rejects the history it already holds.
	breaks loudly	Anything that selects it fails. The failure is loud, immediate, and lands on whoever reads it rather than whoever removed it.
	BREAKS SILENTLY	To a reader this is a drop and an add at the same time. Selects on the old name fail, and any `SELECT *` pipeline quietly starts carrying a new column full of the values the old one used to hold.
	compatible	Every old value fits in the new type. This is the direction evolution is meant to go.
	BREAKS SILENTLY	Values that fit continue to work. Values that do not are truncated or wrapped, and the row survives with a different number in it.
	BREAKS SILENTLY	The cast produces null rather than an error, so the rows survive, the row counts match, and every amount silently becomes nothing.
	BREAKS SILENTLY	The type is unchanged, the range is plausible, and every check passes. This is the change no schema system can catch, because the schema did not change.

The producer sends `amount` as a string — run through the pipeline model

Published

yes

Reported total

0.00 minor unitssim

True total

934,498.90 minor unitssim

Validity check

FAIL

amount arrived as a string. The cast produced null rather than an error, so every row survived with no value in it.

Enforcing the contract does not make the change safe. It makes it visible — the run fails instead of publishing a table of nulls, and somebody is woken up. That is the trade, and taking it is the point: an outage you can see is cheaper than a number you cannot.

SIMULATEDFORMAT-SPECIFICWhether a given change is tolerated depends on the serialization format and the reader: Avro resolves renames through aliases, Parquet readers match by name, and a JSON consumer notices nothing until a value is missing. The row for `amount` as a string is run through the pipeline model; the rest is a declared taxonomy.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

ENGINE-SPECIFICWhether a failed cast raises or yields null is an engine and function choice: most SQL engines offer both a strict cast and a permissive one, and transformation tooling often emits the permissive form by default. The teaching is identical everywhere; the function name and the default are not, so check which one your models actually generate.
SIMULATEDThe counts and check outcomes in the third section come from the in-repo model at src/de/sim/pipeline.ts, which generates a fixed set of source orders and runs them through the pipeline under this exact fault. They are the output of a deterministic model, not a measurement of any real system.
GENERALThe split between consumers that raise and consumers that coerce is universal, because it follows from whether a reader declares its types. What varies by platform is the ratio between the two populations — a strongly typed ingestion path has few coercing readers, an inference-based lake has almost nothing else.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Backenddeserialization parse-dont-validate transport-validation expand-contract-migration

API Designenum-evolution removing-fields validation-errors error-model

Securityinput-validation

Observabilityalerting

Domains that do not exist yet

— DevOps / Production Engineering owns the deploy that caused this: the producer's release, the rollback that would undo it, and the question of whether data written during the window is rolled back with the code.