The Event-Driven Data Platform
Everything publishes events; consumers subscribe independently. It buys decoupling, replay and many materialisations of one stream — and it moves duplicate, ordering and schema handling from one place into every consumer.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If every system publishes its changes to a shared log instead of being queried, what does that decouple — and what does each consumer now have to solve on its own?
Every downstream that used to ask the source a question: the warehouse loader, the search indexer, the cache invalidator, the fraud model, the metrics pipeline. Each one wants the same thing — a complete, ordered, replayable account of what changed — and each will get a slightly different answer depending on how carefully it was written (Writing Event Consumers).
The unit is one event: one thing that happened, at one time, to one entity. That grain is the platform's contract, and getting it wrong is unrecoverable in a way schema mistakes are not — an event that bundles three state changes cannot be un-bundled by any consumer, and one that omits the prior value cannot have it inferred (What a CDC Event Contains).
Have each consumer query the source system it needs. The warehouse runs a nightly extract, the search indexer polls for changed rows, the cache listens for a webhook. Each integration is simple, obvious and independently debuggable, which is exactly why platforms grow this way.
The source system carries the load of every consumer. Five nightly extracts against the same tables at the same hour is a self-inflicted incident, and it lands on the operational database that also serves customers (Workload Isolation).
- The source system carries the load of every consumer. Five nightly extracts against the same tables at the same hour is a self-inflicted incident, and it lands on the operational database that also serves customers (Workload Isolation).
- Every new consumer requires a change to the producer, or at least the producer's consent, so the producing team becomes an approval queue for other teams' work (Data Ownership).
- Nobody can answer "what did this entity look like in March", because each consumer polled current state and the intermediate states were never recorded anywhere (Event vs Snapshot Modeling).
- The consumers disagree. The search index polled at 03:00, the warehouse extracted at 04:00, the cache was invalidated by a webhook that was retried twice, and three systems now hold three versions of the same entity with no way to say which is right.
- Recovery has no common mechanism: fixing the search index means a full re-crawl of the source, fixing the warehouse means a re-extract, and both compete with production for the same capacity (Replay from the Log).
What is actually happening
- The producer publishes a record of what changed to a durable, partitioned, replayable log, and stops caring who reads it. Consumers subscribe at their own pace and maintain their own position. That inversion — from pull from the source to push to a log that many pull from — is the whole pattern (Kafka as a Log, Not a Queue).
- The log is not a queue. A queue delivers a message and forgets it; a log retains it for a configured horizon so that a new consumer can start from the beginning and an existing one can rewind. Replay is the property that distinguishes this from message-oriented integration and it is the one that matters most here (Retention and Replay).
- Ordering is per partition, not global. Events for one entity stay in order only if they consistently hash to the same partition, which means the partition key is a correctness decision rather than a throughput one (Event Keys and Partition Assignment, Topics and Partitions).
- Delivery is at-least-once in every realistic configuration. Duplicates are normal operation, not an incident, and every consumer must be able to process the same event twice without changing its result (At-Least-Once Delivery).
- The centralising work does not disappear; it is redistributed. In a query-based platform, one loader team solved deduplication, ordering and schema drift once. In an event-driven one, every consumer solves all three, at whatever level of care its team happens to have. That redistribution is the pattern's real cost and it never appears on the diagram (Deduplication).
- Publishing itself is harder than it looks: writing to the database and publishing to the log are two operations that can partially fail, which is the dual-write problem. The correct constructions are to publish from the database's own change log or through a transactional outbox (Change Data Capture, The Transactional Outbox).
One publish, many independent readers
The structural change is easy to draw and easy to underestimate. Before, each consumer had an integration with the source: a query, an extract, a webhook. After, the source has one integration — publishing what changed — and every consumer has an integration with the log instead.
The consequence people expect is that adding a consumer no longer requires the producer to do anything, which is true and valuable. The consequence people do not expect is that the producer has also lost the ability to know who depends on its data, which makes every future schema change a leap in the dark unless a contract and a registry fill the gap (Data Contracts).
Read the diagram for what each consumer must now contain. The three boxes attached to each of them — deduplicate, order, resolve schema — are not decoration. They existed once, in the loader, in the old picture. Here they exist once per consumer, and the platform is only as correct as the least careful copy.
What each hop promises, and what it hands to the next one
The value of writing this pattern as a pipeline is that the guarantee column shows exactly where the burden lands. Almost every promise in it is conditional, and every condition is something a person has to build.
Pay attention to the transition between the log and the consumer. That is the only place in the chain where a guarantee gets *stronger*, and it does so only because the consumer added idempotency and event-time ordering itself. Nothing upstream provided them; the log is explicitly entitled to deliver an event twice and to interleave keys from different partitions arbitrarily.
The last row is the one that gets omitted from architecture diagrams. Whatever the consumer materialises becomes an input to something else, and it inherits every weakness above it. A warehouse table built from an at-least-once stream by a consumer with imperfect deduplication is an at-least-once table, and no amount of transactional rigour inside the warehouse changes that (The Data Warehouse).
- 1Transaction commit
The producing service writes state and commits.
guarantees Atomicity and durability of the state change inside that database. Nothing about anything being published.
fails by Business logic that is itself wrong — the strongest guarantee in the chain is about storage, never about meaning.
- 2Capture
Reads the committed change from the database log, or from an outbox row written in the same transaction.
guarantees Every committed change is emitted at least once, in commit order per source. An outbox additionally guarantees the event exists if and only if the state change did.
fails by Falling behind the source log's retention, at which point changes are gone rather than late (CDC Failure Modes and the Retention Deadline).
- 3Publish to log
Appends the event to a topic partition chosen by key.
guarantees Durable retention for the configured horizon; ordering within the partition; readable by any number of independent consumers.
fails by A partition-count change re-hashing keys, so a key's new events no longer share a partition with its own history.
- 4Schema validation
Checks the event against the registered schema and compatibility rule.
guarantees That the structure matches a registered version and that the change satisfies the configured compatibility mode.
fails by Passing a change that is structurally compatible and semantically breaking — a field whose meaning narrowed (Semantic Changes).
- 5Consumer receipt
Reads from its committed offset onward.
guarantees At-least-once receipt of every event after that offset, within retention.
fails by Committing the offset before the work is durable, which converts at-least-once into at-most-once and loses events on a crash (Offsets and Commits).
- 6Consumer processing
Deduplicates by event key, orders by event time, resolves the schema version, applies logic.
guarantees Only what this consumer implemented. Effectively-once processing here requires an idempotent state update or a transactional write to the sink, and is a property of this box alone.
fails by Deduplicating on a field that is not stable across redelivery, so retries create rows rather than replacing them (Deduplication).
- 7Materialised output
Writes the consumer's view — a table, an index, a feature set.
guarantees Whatever the sink provides, bounded above by every row in this table. Atomic publish if built that way.
fails by Being read during a replay, when it legitimately holds a partial history that never existed as a consistent state (Atomic Publish).
Only two rows here can be strengthened without changing anything else: capture (use the database's own log or an outbox) and consumer processing (make it idempotent). Those two are where nearly all of the achievable correctness in this pattern lives.
The checks that make a fan-out platform trustworthy
Monitoring an event-driven platform by watching the log is comfortable and insufficient. The log is usually healthy. The failures live in consumers, they are per-consumer, and an aggregate view of the platform averages them into invisibility.
The rule that follows is that every check here is per consumer, per topic, published as one row each. A platform-wide duplicate rate hides the one consumer that double-counts. A platform-wide lag number hides the one partition that is stuck. The dashboard for this pattern is a table with a row per consumer, not a set of gauges (The Data Quality Dashboard).
The last row below is the check most event-driven platforms lack, and it is the one that catches the worst class of failure. Every consumer can agree perfectly with the log while the log is missing a bulk update someone applied directly to the database. Reconciling the log against the originating system is the only way to see that, and it requires the source to still be reachable — which is a reason to build it early rather than after the first incident (Reconciliation).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Consumer lag per group and per partition | Every consumer is keeping up with the stream it subscribes to. | A stopped consumer, a stuck partition, a poison event blocking progress, a consumer that cannot keep pace with production rate. | A consumer that is perfectly current and processing every event incorrectly. Lag is a liveness signal and says nothing about output correctness. |
| Retention headroom: consumer position versus oldest retained offset | Nothing will be deleted before this consumer reads it. | The silent-gap failure, where retention expires past a lagging consumer and it resumes from a later offset with no error. | A consumer that is within retention and has already skipped events for another reason, such as an offset reset during a deployment. |
| Duplicate rate on the business key in each materialised output | At-least-once delivery is not visible downstream. | Broken idempotency in one consumer, a replay that appended instead of merging, a deduplication key that is not stable across redelivery. | Duplicates whose key differs — the same order re-emitted with a new event id looks like two distinct orders to every check that trusts the id. |
| Per-consumer completeness against the topic for a closed period | This consumer materialised everything the log published. | Dropped events, filtered-out categories, a schema version the consumer silently discarded, a partition never assigned. | Anything about periods still open, and anything wrong in the values of rows that are all present. |
| Reconciliation of the log against the source database | The log is a complete account of what actually happened. | Capture gaps, changes applied outside the application, a connector restart that skipped a range, DDL that the capture could not interpret. | Changes the source itself never recorded, and any period where source and log are wrong in the same way because they share a bug. |
The first four checks compare consumers against the log. Only the fifth asks whether the log is right, and it is the one that catches the failure every other check is structurally unable to see.
Retention semantics, offset management, ordering keys, tiered storage for long horizons and the availability of a managed schema registry all differ substantially between log products and change between versions. Treat the architectural properties above as stable and verify the specific retention, ordering and compatibility behaviour against current documentation for the version you would actually run.
How to build it
Most important first.
- Derive events from the source of truth rather than emitting them beside it. CDC from the write-ahead log or an outbox table written in the same transaction as the state change both guarantee that an event exists exactly when the state changed; an application-level publish next to the commit does not (CDC vs Polling).
- Design the event as a contract, not as a serialised row. Name it for what happened, include the identifier, the event time, a version and enough context that a consumer does not have to call back to the source. An event that requires a lookup to be useful has recreated the coupling it removed (Data Contracts, Naming Events).
- Choose the partition key for ordering correctness first. Per-entity ordering requires that every event for an entity lands in the same partition, and it survives only until someone changes the partition count (CDC Ordering and Transaction Boundaries).
- Register schemas and enforce compatibility at publish time. In this pattern the producer cannot know its consumers, so a compatibility rule enforced centrally is the only thing standing between a field rename and a dozen silent breakages (Schema Registry, Backward Compatibility).
- Give consumers shared machinery for the three problems they all have — deduplication by event key, ordering by event time with a stated lateness policy, and schema resolution. If each team writes its own, the platform has as many correctness levels as it has consumers (Data Platform Engineering).
- Set retention as a recovery-window decision. The horizon you retain is the horizon from which any consumer can be rebuilt, and that is a much more useful framing than a storage argument (Retention and Replay).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Durability and replay within the retention horizon. Outside it, the events are gone and the only recovery path is the source system, if it still holds the state (Retention and Replay).
- Ordering per partition only. Two events for different keys have no defined relative order, and two events for the same key have a defined order only while that key maps to one partition (Topics and Partitions).
- At-least-once delivery to each consumer group. Effectively-once *processing* is achievable per consumer, and only by making the state update or the output write idempotent or transactional — it is a property the consumer builds, never one the log provides (Exactly-Once: Input Consumption, State Update, Output Write).
- No guarantee that two consumers derived the same thing from the same stream. That is the characteristic divergence of this pattern and nothing in the infrastructure prevents it (Two Dashboards, Two Numbers).
- No guarantee of completeness relative to the source. If the capture missed a change — a bulk update applied outside the application, a connector gap during a failover — the log is confidently, silently incomplete (CDC Failure Modes and the Retention Deadline).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check this pattern needs first is per-consumer completeness against the log: for a closed period, count events published to the topic and count records materialised by each consumer, and alert per consumer on divergence (Reconciliation).
- Add a duplicate rate check per consumer on the business key. Duplicates are expected in delivery and must not be visible in output; a rising duplicate rate in a materialised table means that consumer's idempotency is broken while every other consumer looks fine (Duplicate Rows).
- Both miss the case where the log itself is incomplete relative to the source — every consumer agrees with the log and the log is wrong. Only a reconciliation against the originating database catches that, and it is the check most event-driven platforms do not have (CDC Failure Modes and the Retention Deadline).
- The pattern removes scheduling latency: a consumer can process an event as it arrives rather than waiting for a window to close. What it does not remove is the consumer's own processing latency, its batching, or the downstream table's publish cadence.
- Freshness becomes per-consumer and highly variable. The cache updates in seconds, the search index in minutes, the warehouse when its micro-batch commits. Publishing a single platform freshness number under this pattern is actively misleading (Freshness Monitoring).
- Consumer lag — the distance between the newest event in the log and the consumer's position — is the freshness signal that matters, and it is the one metric that means the same thing for every consumer (Offsets and Commits).
- Replay changes freshness in a way that surprises people: a consumer rebuilding from the beginning of the log is maximally *behind* while it catches up, and any downstream reading it during that window sees partial history rather than stale history, which is worse (Atomic Publish).
- A published event is a public interface with an unknown consumer list, so its compatibility rules are stricter than an internal table's. Adding an optional field is safe; removing or retyping one breaks consumers you cannot enumerate (Schema Evolution).
- Consumers evolve at different speeds and the log holds history written under old schemas. A replay therefore reads events in several schema versions, and every consumer needs a resolution strategy for versions it has never seen (Forward Compatibility).
- The most damaging evolution is semantic:
order.status = "complete"starting to mean something narrower after a product change. Every schema check passes, every consumer keeps working, and every derived metric shifts (Semantic Changes). - Changing the partition key or the partition count is a correctness event, not a scaling operation. Existing keys re-hash, and a key's new events can land in a different partition from its old ones — losing order against its own history (Event Keys and Partition Assignment).
- Recovery is the pattern's best feature: reset a consumer's position, rebuild its output from the log, and swap. It requires no coordination with the producer and no load on the source system (Replay from the Log).
- It is bounded absolutely by retention. A consumer that needs to rebuild from before the horizon has no path through the log, and must bootstrap from a source snapshot and then stitch the stream onto it — the same construction CDC uses on first run (Snapshot and Stream: the Bootstrap Problem).
- Rebuild into a new output and switch readers, rather than truncating the live one. During a replay the output is legitimately partial, and consumers reading it will see a version of history that never existed (Atomic Publish).
- Idempotent consumers make replay safe; non-idempotent ones make it a duplication event. This is the single property to verify before promising anyone that replay is available (Idempotent Data Pipelines).
What can go wrong
- A consumer falls behind, retention expires past its position, and it silently skips a range of events rather than failing — a gap with no error attached (Offsets and Commits).
- A partition-count change re-hashes keys and breaks per-entity ordering for every key that moved, producing "latest state" that is an older state.
- One consumer's deduplication is subtly wrong, so exactly one downstream table double-counts while every other consumer of the same topic is correct — which makes the incident look like a transformation bug (Duplicate Rows).
- The producer publishes beside the transaction rather than from it, so a crash between commit and publish loses an event permanently and no downstream can detect its absence (The Dual Write Problem).
- Poison events block a partition: one malformed record that a consumer cannot process halts progress for every key in that partition (A Dead-Letter Queue Is a Workflow, Not a Bin).
- The mitigation failing: shared consumer machinery exists but is optional, so the two teams under deadline pressure wrote their own and the platform has three deduplication semantics.
- "Event-driven means real-time." It removes scheduling latency and says nothing about consumer processing latency, batching or downstream publish cadence. A platform can be fully event-driven and serve hour-old tables (Batch vs Streaming Ingestion).
- "The log guarantees ordering." It guarantees ordering within a partition. Across partitions there is none, and within a partition it survives only while the key mapping is stable (CDC Ordering and Transaction Boundaries).
- "Duplicates mean something is broken." At-least-once delivery is the normal configuration. Duplicates in the *log* are expected; duplicates in a consumer's *output* are the bug (Deduplication).
- "Consumers are decoupled, so they can each do their own thing." They are decoupled operationally and coupled semantically: they all derive from the same events, and two consumers deriving revenue differently produce a divergence nobody owns (The Metrics Layer).
- "We publish events, so we have a data platform." A log of events is an integration substrate. Modelling, quality, catalog, governance and serving are still entirely ahead of you (Data Platform Engineering).
- Events carry the classification of the fields inside them, and a topic that fans out to many consumers is the widest access surface in the platform. Classify at publish time and keep sensitive fields out of general-purpose topics rather than relying on consumer discipline (Data Classification, PII in Pipelines).
- Deletion requests are structurally hard here: an immutable append-only log is designed not to forget. The workable constructions are keeping identifiers out of the log and holding them in a separately deletable store, or tokenising them so deleting the token map renders the log records non-identifying (Deletion Requests, Data Masking, Tokenisation & Encryption).
- Retention has two opposing owners: recovery wants it long, privacy wants it short. That conflict should be settled explicitly per topic rather than by a platform-wide default nobody argued about (Data Retention).
Operating it
- Consumer lag per group and per partition. Per-partition matters because a single stuck partition is invisible in an aggregate lag number (The Backlog Arithmetic: Four Levers and a Drain Time).
- The distance between the oldest retained offset and each consumer's position — the remaining headroom before retention silently eats a consumer's unread events.
- Schema-registry rejections per producer, which is the leading indicator that a breaking change was attempted; and successful registrations of new versions, which is the leading indicator that consumers are about to see something new (Schema Registry).
- Duplicate rate on business keys in each materialised output, tracked per consumer rather than platform-wide (The Data Quality Dashboard).
- Per-consumer completeness against the topic for closed periods, published as one row per consumer so a single bad consumer is visible (Pipeline Metrics).
- At 10x event volume, partitioning and consumer parallelism carry it; the constraint moves to key skew, where one hot entity saturates a single partition regardless of how many you added (Hot Keys: When Aggregate Metrics Hide a Saturated Node, Data Skew).
- At 10x consumer count, the ordering and duplicate problems multiply because each consumer solves them again, and shared machinery stops being a nicety (Data Platform Engineering).
- At 10x topic count, discovery and ownership dominate: nobody can find which topic carries the authoritative account of an entity, and several topics claim to (The Data Catalog, Source of Truth).
- Retention scales with the recovery promise rather than with throughput, so a longer horizon at constant volume is a linear storage increase and a constant operational one.
- Retained bytes in the log, driven directly by the retention horizon you promised. Retention is the recovery window, so this cost is literally the price of the rebuild guarantee (What Actually Drives Data Platform Cost).
- Duplicated derivation: several consumers computing the same enrichment from the same stream is normal in this pattern and is pure repeated compute. It is the cost that motivates a shared curated stream sitting between the raw log and the consumers (Compute Waste).
- Consumer compute is continuous rather than scheduled, so it is paid whether or not events are flowing. A low-volume topic with a dedicated always-on consumer is the most common quiet waste here (Cost vs Freshness).
- Replay is a cost spike by design — rebuilding a year of state reads a year of log and writes a year of output, concentrated into hours (Reprocessing vs Retrying).
- Decoupling producers from consumers costs a shared correctness burden. What one loader team solved once now has to be solved by every consuming team, and the platform's reliability becomes the minimum across all of them.
- Replay is a superb recovery property and it is paid for continuously in retained bytes, whether or not you ever use it.
- Events as contracts make producer changes safer for consumers and harder for producers, which is the trade being made deliberately — the friction is moved to the party who can see the whole picture at change time.
Fan-out — one log, many consumers
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe inversion from consumers pulling the source to producers publishing to a replayable log, and the consequence that each consumer inherits duplicate, ordering and evolution handling, hold for any log-based substrate regardless of product.
- BROKER-SPECIFICKafka orders per partition and retains by time or size with consumer-managed offsets; Kinesis orders per shard and caps retention differently; Pub/Sub gives no ordering at all unless an ordering key is set and acknowledges per message rather than by offset. Replay ergonomics and the shape of a "consumer group" differ across all three.
- SOURCE-SPECIFICHow faithfully events represent the truth depends on where they come from: a Postgres logical-decoding stream carries every committed change, a MySQL binlog in statement format may not, and application-emitted events carry only what the application chose to emit and only when it did not crash first.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what "at-least-once", "per-partition ordering" and "replay" actually mean across machines, including why a globally ordered log is a much stronger and much more expensive object than a partitioned one.
- — DevOps / Production Engineering owns the deployment question this pattern raises constantly: how a consumer is rolled out without resetting its offsets, and how a replay is run alongside live processing without either interfering with the other.