CDC and Schema Drift
A migration runs on the source at 02:00. Some connectors emit a schema-change event, some silently reshape the payload, some stop. None of them ask you first.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The application team renamed a column last night. What did the CDC pipeline do about it — and how long before anyone downstream found out?
Every model built on the captured table, and the analyst reading the dashboard it feeds. Their requirement is not that the schema never changes — it will — but that a change is visible rather than absorbed silently into a column of nulls (Trusting Data).
The unit here is the payload version: the shape of the change event for one table, valid over an interval of log positions. A captured table does not have a schema, it has a sequence of schemas, and any model reading across the boundary between two of them is reading two different things (Schema Evolution).
Read the after object, select the columns you need, and let unknown fields fall on the floor. It works for every additive change, which is the overwhelming majority of migrations, so it survives for a long time and builds real confidence.
A column is renamed. The old name vanishes from the payload and a new one appears. A model selecting the old name now selects a column that does not exist — or worse, in a permissive layer, selects null for every row, and the dashboard reports zero rather than failing (Breaking Schema Changes).
- A column is renamed. The old name vanishes from the payload and a new one appears. A model selecting the old name now selects a column that does not exist — or worse, in a permissive layer, selects null for every row, and the dashboard reports zero rather than failing (Breaking Schema Changes).
- A column is retyped from integer to decimal. The payload types cleanly at every hop, and a downstream cast that was silently truncating now silently rounds differently. No error, different numbers (Nullability & Defaults).
- A
NOT NULLconstraint is added with a backfill. The backfill is a bulk update, so the stream carries one event per affected row — a flood that looks like every customer changing at once (Volume Anomalies). - A column is dropped. New events omit it, old events in raw still carry it, and every model reading across the boundary sees a column that is populated until a date and null afterwards (Backward Compatibility).
- A new enum value appears in
status. Nothing about the schema changed at all, and aCASEstatement in the transformation quietly routes it to theelsebranch, which was written to mean "cancelled" (Semantic Changes). - The connector itself is upgraded and changes how it encodes decimals or timestamps. Every consumer sees a shape change caused by no source migration whatsoever.
What is actually happening
- A DDL statement is written to the source log like anything else, but what a connector does with it varies by design. Three behaviours exist in the wild and you must know which yours has: emit a schema-change event, silently reshape subsequent payloads, or fail.
- Emitting a schema-change event is the honest behaviour: consumers receive a marker at the exact log position where the shape changed, and can act on it. Silently reshaping is the dangerous one, because the payload remains valid and the meaning of the stream changed without anything saying so.
- Failing is the safest and the least popular, because it turns a data-quality problem into an availability problem at 02:00. Teams routinely reconfigure a failing connector to be permissive and then inherit the silent-reshape behaviour without deciding to.
- The deeper issue is that CDC captures the physical schema of a table the producing team believes is private. There is no interface, no version and no deprecation cycle, because from their point of view no consumer exists (Schema Leakage).
- A schema registry with compatibility rules converts a silent reshape into a rejected publish, which is the mechanism that gives the boundary teeth. Compatibility is checked structurally, so it catches renames, drops and retypes and catches nothing semantic (Schema Registry).
- Raw retains both shapes forever, which is what makes recovery possible: a model can be rewritten to read the old and the new shape and re-run over history (Keeping Raw History: The Recovery Position and the Liability). The organisational mechanism that prevents rather than detects is a data contract — an explicit agreement naming captured columns and their meaning, so a migration touching them is a conversation rather than an incident (Data Contracts).
One migration, five consumers, three of them silent
The change below is entirely reasonable from the source team's point of view. amount_cents was a bad name once a second currency appeared, amount_minor is better, and the migration renames it and adds a nullable discount_minor. It passes review, it ships at 02:00, and nothing in the application breaks.
What it does downstream is the point. The rename removes a field and adds one, which is structurally identical to a drop plus an add — and a permissive pipeline will treat it as exactly that, producing null for every row of a column that a revenue model sums. The addition is harmless. The rename is not, and neither has any visible difference at the moment it lands (Breaking Schema Changes).
Read the silent column. Two of the five consumers fail loudly, which is the good outcome — someone is paged, the cause is obvious, the fix is an hour. Three fail quietly, and one of those three is the executive revenue dashboard, which will report a plausible smaller number until someone questions it (The Pipeline Succeeded. The Data Is Wrong.).
- order_id BIGINT
- status TEXT
- amount_cents INTEGER
- currency CHAR(3)
- created_at TIMESTAMPTZ
- updated_at TIMESTAMPTZ
- order_id BIGINT
- status TEXT
- amount_minor INTEGER
- discount_minor INTEGER NULL
- currency CHAR(3)
- created_at TIMESTAMPTZ
- updated_at TIMESTAMPTZ
change Rename amount_cents to amount_minor; add nullable discount_minor. Shipped as one migration at 02:00, with no notification to any data consumer because none was known to exist.
| Consumer | Effect | How it shows up |
|---|---|---|
| Strict boundary check on the raw layer | Rejects the payload: expected column amount_cents absent. The pipeline stops and someone is paged with the exact cause. | Loudly — it raises |
| Staging model selecting named columns | Fails to compile — the column does not exist. Loud, immediate, attributable. | Loudly — it raises |
| Permissive raw loader with schema inference | Accepts both shapes. Rows before the migration have amount_cents and null amount_minor; rows after have the reverse. No error anywhere. | Silently — no error, wrong result |
| Revenue model summing `amount_cents` | Sums null for every row after the migration. Revenue drops to the pre-migration total and stops growing, which looks like a quiet week rather than a fault. | Silently — no error, wrong result |
| BI extract using `SELECT *` | Gains a discount_minor column and a differently named amount. Existing tiles keep rendering; a new one built next month uses whichever column happened to be populated for its date range. | Silently — no error, wrong result |
| Downstream search indexer | Indexes a document with a missing price field. Search results degrade rather than fail, and the degradation is attributed to relevance tuning (Keeping a Search Index in Sync). | Silently — no error, wrong result |
Fail loudly, or absorb it quietly
There are two postures at the boundary and they are genuinely in tension. A strict boundary stops the pipeline on any shape it did not expect; a permissive one accepts whatever arrives and lets the transformation deal with it. Neither is free and the choice is usually made by default rather than by decision.
The argument for strictness is that a data platform's currency is trust, and a stopped pipeline costs freshness while a silent reshape costs correctness. Freshness incidents are visible, attributable and fixed in an hour; correctness incidents are found by a person weeks later and cost the platform's reputation as well as the reprocessing (Data Incidents).
The argument against is real and should not be dismissed: a strict boundary means an upstream team can stop your pipeline at 02:00 with a routine migration, and if that happens often enough the boundary gets relaxed by whoever is on call. The durable answer is not a setting, it is a contract plus an expand-and-contract migration practice upstream, so that breaking changes stop arriving unannounced (Data Contracts).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Expected column set and types per table, asserted at the modelled layer | The payload is the shape this model was written against. | Renames, drops, retypes, and connector envelope changes from an upgrade. | A change in meaning with no change in shape; a value range shift; a new enum member (Semantic Changes). |
| Null rate per column against its own history | Columns are as populated today as they were yesterday. | A rename absorbed as nulls, a dropped column, a truncated large value that arrived as an unchanged placeholder. | A column that was always partly null; a rename to a column that a permissive loader happened to populate from a default. |
| Set membership on low-cardinality columns | The domain of this column is what the transformation assumes. | A new status value, a new currency, a country code convention change — the semantic failures no registry sees (Distribution Tests). | A value that stays in the set and changes meaning; anything on a high-cardinality column, where membership is not assertable. |
| Registry compatibility rule at publish | This payload version is compatible with what consumers were built against. | Structurally incompatible changes, before they reach any consumer (Schema Registry). | Everything semantic, and any change made by a producer that bypasses the registry — which during an incident is most of them. |
| Schema-change events landed and alerted on | The shape of this table changed, at this log position. | The fact of a change, attributably and at the right position, on sources whose log carries DDL. | Everything on a source whose log does not carry DDL, where the change is only inferable from the next differently-shaped row. |
Rows one and three are the pair worth running together: structural assertions catch reshapes and are blind to meaning, membership assertions catch meaning and are blind to shape. A platform running only the first has the more common half of the problem covered and believes it has all of it.
Land whatever arrives, infer the shape per file, let unknown fields through and let missing ones become null. The pipeline never stops for a schema reason.
Land the raw payload untouched and unvalidated, so nothing is ever lost. Then, at the first modelled layer, assert the expected column set and types per table and fail the batch on a mismatch — naming the table, the missing column and the log position where the shape changed.
The two requirements are different and are usually conflated. Raw must accept everything, because a payload rejected at ingest is a payload that no reprocessing can recover. The modelled layer must accept nothing it cannot vouch for, because that is where a null silently becomes a number in a report. Splitting the boundary gives loud failure without any risk of data loss, which neither uniform posture achieves.
The contract, and why it is an organisational mechanism
Every technical defence in this lesson is detection after the fact. The pipeline finds out that a migration happened, at best at the moment it happens, and at worst weeks later. None of them prevent the change, because the producing team does not know a consumer exists (Schema Leakage).
A data contract is the mechanism that fixes the actual problem: it makes the consumer visible. It names the captured table, the columns depended on, their meaning, and the notice required before changing them — and once it exists, a breaking migration becomes a conversation during code review rather than an incident at 02:00 (Data Contracts).
It is worth being honest about what a contract costs. It is organisational effort, it needs an owner on both sides, and it only pays off above a certain number of teams. In a five-person company it is ceremony and a strict boundary check is enough. Across a dozen teams shipping migrations weekly, it is the only thing that turns schema drift from a recurring incident class into a routine coordination (Who Owns Data Quality).
Where the coupling genuinely cannot be tolerated, the answer is not a better contract on a physical table — it is to stop capturing the physical table. An outbox published by the producing service is an interface they own, version and deprecate deliberately, and it is the one arrangement in which a source migration is not automatically your problem (The Transactional Outbox).
- Name the columns. A contract that says "we capture
orders" grants the producing team no useful guidance. One that names six columns tells them exactly which of tomorrow's migrations needs a conversation. - Say what each column means, not only its type.
amount_minoris the order total in the currency's minor unit, excluding tax and after discounts — that sentence is what makes a semantic change detectable by a human, and no registry will ever encode it (Dataset Documentation). - State the notice period and what happens without it. A contract with no consequence is documentation.
- Prefer expand-and-contract in the producing team's migration practice. It converts a breaking rename into an additive add, a dual-write period and a later drop — three safe changes instead of one dangerous one (Expand and Contract Migrations).
- Put impact analysis before the migration, not detection after it. Lineage that answers "what breaks if I drop this column" is worth more than any check that fires once it is gone (Impact Analysis).
- Review the contract when the connector is upgraded, because a connector upgrade changes the payload with no source migration behind it and no contract clause covering it.
How to build it
Most important first.
- Never select
*from a change payload into a model. Name every column you depend on, so a rename fails at the boundary rather than propagating a null (Contract Enforcement). - Register the payload schema per table and enforce a compatibility rule at publish. A change that violates it should be rejected loudly at the boundary, where one team sees it, rather than absorbed quietly into fifty models (Schema Registry).
- Handle schema-change events as first-class data: land them, alert on them, and make them queryable. "When did this table's shape change" is an incident question and should take one query (Metadata: Technical, Operational and Business).
- Prefer expand-and-contract migrations on the source: add the new column, dual-write, migrate consumers, drop the old one later. It converts a breaking change into two additive ones, and it is the mechanism that makes upstream schema evolution survivable (Expand and Contract Migrations).
- Write the contract down for the tables that matter, and keep it short: the columns captured, their types, their meaning, and who to tell before changing them (Data Contracts).
- Assert semantics, not only structure — a test that
statusis in a known set catches the new enum value no schema check can (Data Tests) — and version the model layer against the payload version, so history built under an old shape stays reproducible rather than being retroactively reinterpreted (Model Layering).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- CDC guarantees at-least-once delivery of every committed change, ordered by source log position. It guarantees nothing whatsoever about schema stability — the payload shape is whatever the source table currently is.
- It does not guarantee that a DDL produces a visible event. Whether one is emitted, and whether it is emitted at the right position, is a connector property and a configuration (What a CDC Event Contains).
- It does not guarantee that the connector's own envelope stays stable across upgrades, so a shape change can arrive with no source change at all.
- A schema registry guarantees only structural compatibility against a declared rule. It cannot see that a field's meaning changed, and it is routinely mistaken for a semantic guarantee (Semantic Changes).
- Nothing guarantees consistency between the shape of events already in raw and the shape of new ones. Both exist, both are valid, and any model reading across the boundary must handle both (Backward Compatibility).
- A contract guarantees a conversation, not a mechanism. It is enforced by people unless it is also enforced by a check at the boundary (Contract Enforcement).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Assert the expected column set and types on the payload at the boundary, per table, and fail the batch on a mismatch. This is the check that turns a rename into a page instead of a column of nulls (Contract Enforcement).
- Assert null rate per column against its own history. A rename that produced nulls, a dropped column, and a truncated large value all show up here and nowhere else (The Dimensions of Data Quality).
- Assert set membership on low-cardinality columns — statuses, currencies, country codes — because a new enum value is a semantic change that no structural check can see (Distribution Tests).
- All three miss a retype that preserves shape and changes precision, and all three miss a field whose meaning changed while its values stayed in range. Only a consumer who knows the domain finds those (Two Dashboards, Two Numbers).
- A failing connector converts a schema change into a freshness incident: nothing flows until someone intervenes, which is loud, attributable and recoverable (The Freshness SLO).
- A silently reshaping connector converts it into a correctness incident with no freshness symptom at all. The data stays perfectly fresh and quietly means something different, which is far worse and far harder to detect (The Pipeline Succeeded. The Data Is Wrong.).
- A migration that backfills a column produces a volume spike, and the resulting lag is a freshness symptom whose cause is a schema change — an association worth knowing during an incident (The Backlog Arithmetic: Four Levers and a Drain Time).
- A strict boundary trades freshness for correctness deliberately: it stops the pipeline rather than admitting data it cannot vouch for, and that trade should be an explicit product decision rather than a default.
- This lesson is the evolution case for CDC, and its central asymmetry is that additive changes are usually absorbed while renames, drops and retypes are not (Schema Evolution).
- The producing team's migration process is the real control point. A change that ships through expand-and-contract is survivable by construction; a change that renames in place is not, no matter what the pipeline does (Expand and Contract Migrations).
- Connector upgrades are schema changes with no source migration behind them, and they should be scheduled and validated like one (Running Two API Versions in One Service).
- Semantic drift is the residue that no mechanism catches: an
amountmoving from gross to net, astatusgaining a value, acountryswitching from billing to shipping. The defence is documentation and consumer-side assertions, not a registry (Semantic Changes).
- Recovery from a shape change is a reprocess from raw, which is available exactly because raw retained both shapes. Rewrite the model to read old and new, re-run over the affected range, validate, publish (Reprocessing vs Retrying).
- Bound the re-run to the affected range and write to a location consumers are not reading, then swap. A schema-drift backfill that overwrites the current partition while consumers read it turns one incident into two (Planning a Backfill).
- If the pipeline was permissive and produced nulls for weeks, the affected range starts at the DDL, not at the day someone noticed. Finding that date is why schema-change events belong in a queryable table (Data Lineage).
- If the connector failed and stopped, recovery is ordinary catch-up — provided the position is still inside retention, which a multi-day schema incident can easily outlast (CDC Failure Modes and the Retention Deadline).
- Nothing recovers a semantic change. The values were correct under the old meaning and are correct under the new one, and only a documented cutover date makes history interpretable (Dataset Documentation).
What can go wrong
- A rename absorbed into a column of nulls, discovered weeks later by a finance team rather than by a monitor.
- A retype that types cleanly at every hop and changes precision in the final aggregate.
- A constraint backfill producing a bulk change that saturates the pipeline and looks like a business event (Backpressure).
- A connector configured to be permissive after a failure at 02:00, converting every future breaking change into a silent one.
- A schema registry passing a structurally compatible change whose meaning is different — the registry is working correctly and the data is wrong (Semantic Changes).
- The mitigation fails too: a contract that lists columns and not meanings, so a producing team can honour it perfectly while changing what
amount_minorcounts.
- "The schema registry protects us." It protects the structure. A field whose meaning changed passes every compatibility rule ever written (Semantic Changes).
- "Adding a column is always safe." It is safe for consumers that name their columns. It breaks positional unpacking, it changes
SELECT *output, and it can trigger a backfill that floods the pipeline (Breaking Schema Changes). - "The connector will tell us." Some connectors emit a schema-change event, some silently reshape and some fail. Know which yours does before you rely on being told (What a CDC Event Contains).
- "We can fix it when it breaks." A rename that produces nulls does not break. It reports zero, confidently, until someone senior notices (Stale Dashboards).
- "The source team should just tell us." They will, sometimes, if they know you exist. CDC on a physical table means they have a consumer they never agreed to have, which is a problem to solve with a contract rather than with irritation (Data Ownership).
- A newly added column is captured automatically and enters the platform without review, which is how personal data arrives in a lake by default rather than by decision (PII in Pipelines).
- Column filtering at the connector must be revisited whenever the source schema changes, or an allow-list becomes stale and a deny-list becomes wrong (Data Minimization).
- A retype or rename can change a column's classification without anyone reclassifying it — a free-text field replacing an enum is a different privacy proposition entirely (Data Classification).
Operating it
- A queryable table of schema-change events per captured table, with log position and timestamp. It answers "when did this shape change" in one query during an incident (Metadata: Technical, Operational and Business).
- Null rate per column per day, per captured table. It is the broadest detector of a silent reshape and it costs one aggregate (The Data Quality Dashboard).
- Distinct value count and set membership on low-cardinality columns, tracked over time (Distribution Tests).
- Boundary rejection count: how often the contract check refused a payload. Zero forever usually means the check is not actually running (Quality Alerting).
- Connector version and payload schema version recorded alongside each raw partition, so a shape change caused by an upgrade is attributable (Dataset Documentation).
- At 10x captured tables, per-table contracts stop being writable by hand and the registry plus generated assertions become the only workable mechanism (Schema Registry).
- At 100x consumers, a breaking change's blast radius is only knowable from lineage, and impact analysis before the migration becomes worth more than any detection after it (Impact Analysis).
- More producing teams means more migrations per week, and the probability that one of them is breaking approaches certainty — which is the point at which contracts stop being ceremony (Data Ownership).
- Longer history means more historical shapes to support, and models that read across many boundaries become the least maintainable code in the platform (Model Layering).
- A strict boundary costs pipeline availability during upstream changes and saves the cost of reprocessing weeks of wrong data. The second is almost always larger and arrives later (What Actually Drives Data Platform Cost).
- Handling both shapes in a model costs a permanent branch in the transformation, which accumulates: a table with four historical shapes has four branches nobody dares delete (Compute Waste).
- A constraint backfill costs a full table's worth of change events through the broker and every consumer, which is the same shape as a re-snapshot and is rarely anticipated (Storage Lifecycle).
- Reprocessing from raw after a silent drift costs compute proportional to the undetected interval, which is the strongest cost argument for detecting shape changes at the boundary.
- A strict boundary buys loud failure and costs availability during upstream change. A permissive boundary buys uptime and pays for it in silent wrongness that surfaces weeks later.
- A registry buys structural safety cheaply and buys nothing semantic, which is exactly the gap people fill with unwarranted confidence (Data Contracts).
- Contracts buy coordination and cost organisational effort that only pays off above a certain number of teams. In a small company they are ceremony; across a dozen teams they are the difference between a platform and a weekly incident (Who Owns Data Quality).
- Supporting multiple payload shapes buys uninterrupted history and costs permanent complexity in every model that spans the boundary.
CDC drift lab
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Change | Downstream effect |
|---|---|
| A sink with an explicit column list ignores it forever, and nobody learns the field exists until someone asks for it in a meeting. | |
| The warehouse column stays, filled with nulls from the drop date onward. Every historical average silently changes shape. | |
| Read as a drop plus an add. The data is intact and lands in a new column, and the old one freezes. | |
| Depends entirely on the sink. A cast to the old type may null the values, which is the loudest quiet failure in this domain. | |
| A topic appears, a sink is not configured, and the data accumulates in a broker until retention deletes it. | |
| Partitioning by key breaks: the same row now hashes elsewhere, and its old and new changes can be applied out of order. |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat a consumer of a physical table shape inherits every migration made to it, without notice or versioning, is true of any capture mechanism reading a database directly. It is not true of a published event contract, which is the whole argument for an outbox where the coupling matters.
- SOURCE-SPECIFICHow a DDL appears in the change stream differs sharply: MySQL writes DDL statements into the binary log so a connector can emit an explicit schema-change event at the right position, Postgres logical decoding does not carry DDL and connectors infer the change from the next differently-shaped row event, and MongoDB has no schema to change so drift arrives as documents that simply have different fields. The detection strategy has to be built for whichever of those you have.
- TOOL-SPECIFICWhether a breaking change fails the connector, reshapes the payload silently, or emits a marker is a property of the connector and its configuration rather than of CDC, and the same source can produce all three behaviours under different settings. Establish which behaviour yours has by testing a rename in a non-production copy, not by reading a description.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the migration delivery practice this lesson depends on — how a schema change is reviewed, staged, rolled out and rolled back, and how a consumer outside the deploying team gets told. Expand-and-contract is a delivery discipline before it is a data one.
- — Distributed Systems owns the general form of the compatibility problem: two independently deployed components that must agree on a message shape while neither can be upgraded atomically with the other. The registry rules here are one concrete instance of that.