Forward Compatibility
The consumer ships first: data written under the old schema must still be readable, and still correctly interpreted, by code running the new one. In analytics this is not an edge case — every query over history is this question.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If the consumer upgrades its schema today, can it still read the years of data that were written before the change — and will it interpret them the way it should?
Anyone whose read spans a schema boundary, which in an analytical platform is almost everyone. A dashboard covering thirteen months, a model trained on two years, a finance restatement of last quarter, and — most dangerously — a backfill job running today's transformation code over data written eighteen months ago (Backfills).
The unit is one field, read out of a partition written under an earlier schema version. The question is always about a specific historical range: a field added in March is absent from every record before March, and what the new reader does with that absence is the whole lesson.
Upgrade the consumer and assume the engine will sort it out. Modern formats and warehouses do resolve missing columns to null, and queries do keep running, so the upgrade appears to succeed. The failure is not that the query errors; it is that it does not.
A field is added with a sensible default. Every historical record now reports that default, so two years of orders claim standard shipping because that is what the default said, and nobody can distinguish "standard" from "we did not record it" (Nullability & Defaults).
- A field is added with a sensible default. Every historical record now reports that default, so two years of orders claim standard shipping because that is what the default said, and nobody can distinguish "standard" from "we did not record it" (Nullability & Defaults).
- A backfill runs current transformation code over historical raw data. The code expects a field that did not exist then, reads null, and rewrites two years of a fact table with a column of nulls — replacing history that was correct (What Backfills Break).
- A field is removed from the new schema, and the reader stops projecting it. Historical partitions still contain it, so the data is still there and no query returns it, which is how a column becomes invisible without being deleted (Column-Level Lineage).
- A type is narrowed. New code reads old records whose values do not fit, and a permissive engine returns null rather than raising, so old periods silently lose values while recent ones are fine (Breaking Schema Changes).
- A metric definition is updated in the transformation layer and applied to all history at once, so a number that was published and acted on last quarter now reports differently and neither version is reproducible (Two Dashboards, Two Numbers).
- An old partition written before an enum value existed is read by new code that assumes the value set is complete, and the
ELSEbranch quietly absorbs everything it does not recognise (Enum Evolution: The New Value That Broke Old Clients).
What is actually happening
- This is the same writer/reader relation as the previous lesson with the roles swapped: the writer is old and the reader is new. Data engineering hits this direction far more often than service engineering does, because analytical consumers read history and service consumers mostly read the present.
- The resolution mechanism is a default. When a reader declares a field the writer never sent, something has to fill it: the reader's declared default, a null, or an error. Which of the three you get is a property of the format and the engine, and it is the single highest-leverage configuration choice in this area (Avro).
- A default is a value, and values are indistinguishable from facts once written. This is the mechanism behind the module's quietest failure: a default fills a gap with something plausible, the plausible thing is then aggregated, and the resulting number describes a world that never existed (Nullability & Defaults).
- The reader in this relation is not always a consumer. A reprocessing job is a new reader over old data, so every backfill is a forward-compatibility exercise, and it is one that writes rather than reads — which means a mistake here does not produce a wrong answer, it produces a wrong table (Reprocessing vs Retrying).
- A schema registry enforces this direction under the mode it calls BACKWARD, which compares a candidate schema against previously registered ones and rejects it if it could not read data written under them (Schema Registry).
The consumer ships first, and history does not change to suit it
The previous lesson asked whether an old reader survives new data. This one asks the mirror question, and it is the one that dominates analytical work: a consumer upgrades, and the data it must read was written months or years before the upgrade existed.
Service engineering meets this direction rarely, because a request handler mostly sees traffic produced by the current generation of clients. An analytical query sees every generation at once. That asymmetry is why a data platform needs to be deliberate about this direction in a way that an API team often does not.
The diff below shows a consumer moving from v1 to v2. One field is added and one is dropped. Read the silent column: the additions and removals that would be loud in a service context are all quiet here, because the engine's job is to make a missing column readable and it does that job well.
- order_id: string
- placed_at: timestamp
- amount_minor: integer
- currency: string
- promo_code: string
- order_id: string
- placed_at: timestamp
- amount_minor: integer
- currency: string
- shipping_method: string (default "standard")
change The consumer's model is updated to select shipping_method and to stop selecting promo_code. The producer has been emitting shipping_method only since March. Every partition before March was written without it.
| Consumer | Effect | How it shows up |
|---|---|---|
| Dashboard filtered to the last 30 days | Entirely correct. The range does not span the boundary, which is why this class of bug is almost never caught in testing. | Loudly — it raises |
| Dashboard covering the last 13 months | Every order before March is reported as standard shipping because the default filled it. The shipping-method breakdown is confidently wrong for ten of the thirteen months. | Silently — no error, wrong result |
| Backfill of `fct_orders` over two years | Writes the default into two years of a published fact table, replacing rows that were correct with rows that assert a fact nobody recorded. | Silently — no error, wrong result |
| Promo analysis notebook reading `promo_code` | The column is still physically present in history but the model no longer projects it, so the analysis returns nothing and the analyst concludes the promotion had no participants. | Silently — no error, wrong result |
| Strict reader asserting the exact v1 column set | Fails immediately on the first v2 partition. The only consumer in this table that gets an error, and the only one that will be fixed the same day. | Loudly — it raises |
Defaults are the mechanism, and the mechanism lies
coalesce over a column absent from an older file even reaches your query depends on the engine and table format: some resolve the missing column to null and let the query proceed, others refuse the read until the table schema is evolved explicitly. The teaching point survives either way, but the exact SQL that reproduces it does not transfer between engines.When a reader declares a field the writer never sent, three things can happen: an error, a null, or a default. The first is the only one that tells you something. The second is honest. The third invents a fact.
The distinction is not academic, because the invented fact is then aggregated. A default of "standard" on a shipping-method column produces a bar chart in which the pre-March period is a solid block of standard shipping — a shape so plausible that nobody questions it. Had the same reader produced null, the chart would show a gap, and a gap is a question.
The SQL below is how a careful read across a version boundary actually looks. It does not pretend the boundary is not there; it makes it a column. That single extra column converts every downstream aggregate from "wrong for old periods" into "correctly refusing to answer for old periods", which is the only honest answer available.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Null rate per column, per historical era | A field is populated in the periods where the producer was actually sending it. | A field added mid-history, a producer that stopped populating a column, a narrowed type nulling old values. | Any field filled by a default — its null rate is zero and it looks perfectly healthy while being entirely invented. |
| Distinct-value count per era on low-cardinality columns | The value set for an era looks like the value set you expect for that era. | A default flooding history with one value; an enum whose membership changed at a boundary. | A default that happens to match the dominant real value, which is exactly what a well-chosen default usually is. |
| Recompute a closed period and diff against what is published | Today's code, run over yesterday's data, still produces yesterday's published answer. | Every silent restatement: metric redefinitions, compatibility drift, backfill logic that reads history differently from how it was originally read. | A period that was already wrong when it was published — the diff is zero and both sides are wrong (Reconciliation). |
| Schema version recorded and asserted per partition | Every partition can say which shape it was written under. | The ambiguity between "absent then" and "missing now", which is otherwise unresolvable. | Everything about meaning. A partition can be schema version 2 and have been produced by a definition that changed the same week (Semantic Changes). |
The third row is the strongest check in the module and the one that is almost never implemented, because it costs a recomputation of data everyone believes is finished.
1-- WRONG: the default makes ten months of history claim a shipping method2-- that was never recorded. Nothing errors; the chart looks fine.3select4 date_trunc('month', placed_at) as month,5 coalesce(shipping_method, 'standard') as shipping_method,6 sum(amount_minor) as revenue_minor7from stg_orders8group by 1, 2;9 10-- RIGHT: keep "not recorded" distinguishable from "recorded as standard",11-- and expose the boundary so a consumer can see it.12select13 date_trunc('month', o.placed_at) as month,14 case15 when o.placed_at < date '2026-03-01' then 'not recorded'16 when o.shipping_method is null then 'unknown'17 else o.shipping_method18 end as shipping_method,19 sum(o.amount_minor) as revenue_minor20from stg_orders o21group by 1, 2;22 23-- Better still: put the boundary in metadata rather than in a literal date,24-- so the query does not have to know when the producer changed.25select26 date_trunc('month', o.placed_at) as month,27 case28 when p.schema_version < 2 then 'not recorded'29 when o.shipping_method is null then 'unknown'30 else o.shipping_method31 end as shipping_method,32 sum(o.amount_minor) as revenue_minor33from stg_orders o34join partition_metadata p using (partition_key)35group by 1, 2;Three categories, not two: recorded-as-standard, recorded-as-unknown, and never-recorded. The first query collapses all three into one and can never be un-collapsed once its output has been published.
Every backfill is this question, with write access
The most consequential new reader of old data is not a dashboard. It is a reprocessing job, running today's transformation code over a historical range and writing the result into a table people already trust (Reprocessing vs Retrying).
A dashboard that misreads history produces a wrong picture that disappears when you fix the query. A backfill that misreads history produces a wrong table that persists until someone notices and can only be undone if the previous version still exists. The compatibility question is identical; the blast radius is not.
This is why the discipline around backfills is disproportionate to their apparent difficulty. The steps are not bureaucracy: recompute into a side location, diff against what is published for the same closed period, require a human to explain every difference, and only then publish atomically (Atomic Publish).
- A field added after the range you are reprocessing will read as null or as a default for the whole range. Decide which, deliberately, per field (Nullability & Defaults).
- A metric definition that changed since the range was originally computed will restate published numbers. That may be correct and it is never automatic — it needs an announcement (The Metrics Layer).
- A dimension that has changed since then will join differently unless it is history-preserving, so a backfill can re-attribute old facts to a customer's current segment (Slowly Changing Dimensions).
- A source that has been mutated since the range was ingested cannot be reprocessed from the source at all — only from your own immutable raw (Keeping Raw History: The Recovery Position and the Liability).
- A backfill that appends rather than replaces is not a repair; it is a second copy of a period that was already there (What Backfills Break).
Point the existing job at a two-year range and let it write into the published table. It succeeds, takes several hours, and the table now reflects current logic for all of history. Row counts are unchanged, so nothing alerts.
Write the recomputation to a side location. For a closed period, diff it against what is currently published on row count, on the summed measures and on per-column null rates. Require someone to explain every difference before the publish, then swap atomically and keep the previous version until the next cycle.
New code reading old data is a compatibility relation, not a repeat of a previous run. Any difference between the recomputation and the published table is either the bug you set out to fix or a second change you did not intend, and the only way to tell them apart is to look at them before they are the only copy you have. The diff also costs a period's worth of compute rather than a quarter's worth of restatement.
How to build it
Most important first.
- Give every added field an explicit, defensible answer for what it means in history — and prefer null meaning "not recorded" over a plausible default, because null is honest and a plausible default is a fabrication that aggregates (Nullability & Defaults).
- Record the schema version with the data so a reader can tell whether a null means "absent then" or "missing now". Without it those two facts are stored identically and cannot be separated afterwards (Metadata: Technical, Operational and Business).
- Treat every backfill as a compatibility test: run the new code over an old range into a side location, compare with what is currently published, and require someone to explain every difference before publishing (Validating a Backfill Before You Publish).
- Keep transformation logic versioned alongside the data it produced, so a restatement is a deliberate, dated event rather than an accident of redeployment (dbt Concepts).
- Read history through a view that makes the version boundary explicit — coalescing old and new field names, casting both to a common type, and flagging the periods where a field did not exist (Model Layering).
- When a metric definition changes, publish it as a new column or a new metric rather than redefining the existing one in place. Two clearly named metrics are cheaper than one metric whose meaning depends on when you asked (The Metrics Layer).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- When this direction holds, it guarantees the new reader can parse historical data. It guarantees nothing about whether the values it fills in are true.
- A format-level default guarantees a value will be present. That is the opposite of a guarantee that the value is known, and conflating the two is how history gets fabricated (The Dimensions of Data Quality).
- Nothing guarantees that a field absent from history is *knowable*. If the producer never recorded shipping method before March, no amount of schema work recovers it — the information was never captured (Source of Truth).
- A registry check on this direction covers only the schemas registered with it. Historical files written before the registry existed, or written by a producer that bypassed it, are outside the guarantee entirely.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is a history read test: run the new reader against a sample from each historical schema era and assert that field-level null rates match what you expect for that era (Data Tests).
- It misses defaults that are plausible. A field defaulted to "standard" has a null rate of zero and looks perfectly healthy — the check confirms the field is populated and cannot tell you the population is invented.
- Pair it with a published-versus-recomputed comparison before any backfill publishes. Any difference between what is live and what the new code produces for a closed period is either a bug you just found or a restatement you are about to make, and both need a decision (Validating a Backfill Before You Publish).
- This direction is what buys consumer independence: a consumer can upgrade without waiting for the producer, and without a coordinated window in which data is paused.
- It is also what makes reprocessing possible at all. A platform where new code cannot read old data can only ever recompute forward, which means every bug found today is permanent for every period before today (Keeping Raw History: The Recovery Position and the Liability).
- The latency cost appears at the boundary: a query spanning a schema change has to reconcile two shapes, and doing that in a view rather than in physically rewritten data trades storage for query time.
- Every schema change adds a new era to history, and eras accumulate. A dataset five years old with quarterly changes has a substantial number of shape boundaries in it, and a query spanning all of them is doing implicit reconciliation at every one.
- The practical answer is periodic normalisation: rewrite history into the current shape, once, deliberately, with the old shape retained as raw. That converts a permanent read-time cost into a one-off write cost (Reprocessing vs Retrying).
- Normalisation is itself a forward-compatibility event and the most dangerous kind, because it writes. Validate before publishing, and never do it during an incident.
- If a new reader has misread history without writing, recovery is free: fix the reader and re-read. Nothing was harmed.
- If a new reader has misread history and written — a backfill — recovery requires the previous version of the table, which exists only if the publish was versioned or snapshotted (Snapshot Tables).
- If neither the previous table version nor the raw data exists, recovery is not possible and the history is simply the new, wrong version. This is the case that argues most strongly for immutable raw (Keeping Raw History: The Recovery Position and the Liability).
- Recovering the distinction between a default and a real value after the fact is generally impossible, which is why the design advice is to prefer null at write time rather than to plan to untangle it later.
What can go wrong
- A plausible default filling history and being aggregated as fact.
- A backfill running new code over an old range and overwriting correct history with nulls (What Backfills Break).
- A metric redefinition applied to all history silently, so previously published numbers no longer reproduce (Two Dashboards, Two Numbers).
- A permissive cast producing null for old records that do not fit a narrowed type, so only the historical half of a query is wrong.
- A registry check that passes because it compares only registered schemas, while the actual historical files were written under a shape the registry never saw.
- Normalisation of history performed during an incident, under time pressure, without a comparison step.
- "Forward compatible means new consumers keep working with new producers." It means a new reader can handle data written by an old writer. And a registry configured with a mode called FORWARD enforces the opposite direction from what this lesson describes (Backward Compatibility).
- "The query ran, so history is fine." A query over a range where a field did not exist returns nulls or defaults and runs perfectly. Success is not evidence; a per-era null-rate chart is.
- "A default is safer than a null." A default is more convenient and less true. Null propagates through aggregates in ways people notice; a default propagates in ways they do not (Nullability & Defaults).
- "A backfill just re-runs the DAG." A backfill runs today's code against yesterday's data, which is precisely this compatibility question, with write access (What Backfills Break).
Operating it
- Null rate per column per historical era, not just per recent day. A column that is 100% null before a date and 0% after it is a schema boundary, and seeing that on a chart is how you discover boundaries nobody documented (The Data Quality Dashboard).
- Schema version recorded per partition, queryable, so "which shape was this written under" is answerable in seconds (Metadata: Technical, Operational and Business).
- A diff between the currently published table and a recomputation of a closed period, run on a schedule rather than only before backfills — it detects logic drift as well as compatibility problems (Reconciliation).
- Row counts and summed measures for closed historical periods, tracked over time. A closed period whose numbers move is a restatement, whether or not anyone intended one (Volume Anomalies).
- At 10x history, the number of schema eras in a range grows and the read-time reconciliation stops being negligible; normalisation becomes a scheduled maintenance activity rather than an option.
- At 100x history, rewriting all of it stops fitting in any window, and the only viable approach is to normalise a recent window and serve older data through an explicitly versioned read path (Full Refresh vs Incremental).
- Consumer count does not change this direction much — history is shared, so the reconciliation work is done once per query rather than once per consumer, and a materialised normalised view amortises it across everyone.
- Reading across schema eras through views costs query time at every read, proportional to how many boundaries the range spans (Scan Cost).
- Normalising history costs a full rewrite of retained bytes, once. It is the classic trade of a large one-off write cost against a small permanent read cost, and which side wins depends on how often the range is queried.
- Recording schema version per partition costs almost nothing and is the cheapest insurance in this module.
- Preferring null over a plausible default is more honest and strictly more annoying: every downstream consumer must now handle a null they would not have seen, and some of them will handle it by coalescing to the same plausible value one layer further down.
- Views that reconcile across eras keep history readable and make every query slower and harder to reason about. Physically normalising history is faster to query and destroys the evidence of what the data actually looked like unless raw is retained.
- Enforcing this direction constrains the consumer as much as the other direction constrains the producer: you cannot narrow a type or drop a field from the reader without deciding what that does to every historical range.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe relation between an old writer and a new reader is universal, and it is unusually load-bearing in analytics because analytical consumers read years of history while service consumers mostly read the present. The vocabulary is not universal: this direction is what a Confluent-style registry calls BACKWARD.
- FORMAT-SPECIFICAvro supplies the reader's declared default for a field the writer omitted, which makes the outcome defined and configurable per field; Parquet leaves a missing column to the engine, which usually produces null; CSV has no concept of a missing field at all, so an added column shifts every position after it.
- ENGINE-SPECIFICWhether reading a column absent from an older file yields null, an error or a default is decided by the query engine and by table-format metadata rather than by the file. The same historical partition can therefore read differently through two engines pointed at the same storage.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the rollback question this lesson raises: if the consumer is rolled back after it has written under the new interpretation, what happens to the data it produced in between.