Change Data Capture

Reading a database's own change log instead of asking it questions. What CDC gives you that polling cannot, what it costs the source, and every way it silently loses or reorders changes.

Change Data Capture
▶ lab

Reading committed changes out of the database's own transaction log, so downstream systems learn what happened instead of repeatedly asking what is true now.

Q · A row in `orders` changed at 03:14. How does a downstream system find out about that change — without interrogating the database on a loop, and without missing the change entirely?
CDC vs Polling
▶ lab

One asks the database what is true now, on a loop. The other observes what the database committed. The difference is not speed — it is which changes are structurally invisible.

Q · A nightly job selects every row where `updated_at` is newer than the last run. Which real changes does that query structurally fail to see, and when is it still the right answer?
What a CDC Event Contains
▶ lab

An operation, a before image, an after image and source metadata. Which of those you actually receive is decided by the source's configuration, not by CDC.

Q · A CDC record arrives describing an update to one order. What is inside it, what is deliberately absent, and which parts can you rely on being there?
CDC Ordering and Transaction Boundaries
▶ lab

The source log has a total order and a transaction boundary. Publishing splits both, and every consumer that joins two tables inherits the consequences.

Q · One transaction updated `orders` and `order_items` together. Downstream, why can a consumer see one of them and not the other — and for how long?
Snapshot and Stream: the Bootstrap Problem
▶ lab

CDC starts from now. Everything that existed before now has to be read separately and stitched to the stream without a gap and without a duplicate that a later ordering guard cannot resolve.

Q · The connector starts today against a table with four years of rows in it. Where do the four years come from, and how do you join them to the live stream without losing a change or double-counting one?
CDC Failure Modes and the Retention Deadline
▶ lab

A consumer falls behind. Whether that is an inconvenience or an unrecoverable data loss is decided entirely by whether the connector's position is still inside the source's retained log.

Q · The connector has been down since Friday and it is now Monday. Can you replay what you missed — and how would you find out before you promise someone that you can?
CDC and Schema Drift
▶ lab

A migration runs on the source at 02:00. Some connectors emit a schema-change event, some silently reshape the payload, some stop. None of them ask you first.

Q · The application team renamed a column last night. What did the CDC pipeline do about it — and how long before anyone downstream found out?