Data Engineering and Distributed Systems
A pipeline does not choose its guarantees. It inherits them from the weakest hop, and most pipeline bugs are a distributed-systems property arriving where nobody expected it.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which distributed-systems guarantees does a data pipeline actually depend on, and what breaks when they do not hold?
The analyst who assumes a fact table contains each order once, in the order the orders happened. Neither of those is promised by anything in the chain unless someone built it, and the analyst has no way to tell from the table.
The unit here is one message with an offset in one partition. Everything a pipeline can conclude about ordering, duplication and completeness is a statement about that unit and the partition it sits in — not about the topic, not about the dataset, and not about the business entity the message describes.
Treat the broker and the processing framework as reliable transport: producers put events in, consumers get events out, and the interesting work is the transformation in the middle. This is exactly the mental model the tools' quick-start guides encourage, and for a single-partition demo it is accurate.
The same order appears twice in the fact table. Nothing is broken: the producer retried after a timeout it could not distinguish from a failure, delivery is at-least-once, and the pipeline had no idempotent merge (At-Least-Once Delivery, Deduplication).
- The same order appears twice in the fact table. Nothing is broken: the producer retried after a timeout it could not distinguish from a failure, delivery is at-least-once, and the pipeline had no idempotent merge (At-Least-Once Delivery, Deduplication).
- A customer's status history is out of order. The topic's partition count was increased, so that customer's key hashes to a different partition than it used to, and its new events are ordered against strangers rather than against its own past (Topics and Partitions, Event Keys and Partition Assignment).
- A daily aggregate is short, then correct the next day, then short again. Events are being assigned to windows by arrival time while the business cares about event time, so anything that arrived late landed in the wrong day (Event Time, Late Events).
- A stream job restarts and re-emits an hour of output. Its state was restored from the last checkpoint, and the sink is an append-only table with no key, so the replayed hour is added rather than replacing (Checkpointing, Exactly-Once: Input Consumption, State Update, Output Write).
- Consumer lag grows for six hours during a traffic spike, then the retention window passes and the unread events are deleted. The pipeline recovers, the dashboard fills in, and six hours of history are permanently absent (Retention and Replay, Consumer Groups and the Parallelism Ceiling).
- A job that joins two streams produces fewer matches than expected, because one side's watermark advanced past the other's and the buffered state for the unmatched keys was released (Stream Joins, Watermarks).
What is actually happening
- Delivery semantics are a property of the whole path, not of a product. At-most-once loses, at-least-once duplicates, and there is no third option available from transport alone. What a pipeline calls "effectively once" is at-least-once transport plus an idempotent or transactional sink — the duplicates still happen, they just stop being observable (Exactly-Once: Input Consumption, State Update, Output Write).
- Partitioning is what buys ordering and what limits it. A partitioned log gives a total order within a partition and no order at all across partitions, so ordering is only useful if the partition key is the entity whose order matters. This is the same reasoning as choosing a shard key, one abstraction up (Topics and Partitions, Event Keys and Partition Assignment).
- Time is two things. Event time is when it happened, according to the producer's clock; processing time is when the pipeline saw it. In a distributed system these diverge without bound, and every windowed aggregate must state which one it means (Event Time, Processing Time, Ingestion Time).
- A watermark is a bet, not a fact. It is the pipeline asserting "I believe I have seen everything up to time T", so that windows can close and state can be released. Being wrong is not an error condition — it is the normal case, and the allowed-lateness setting is how much wrongness you are prepared to absorb (Watermarks).
- Checkpoints are what make a stateful stream job restartable. They snapshot operator state and input positions together, so recovery resumes from a consistent point rather than from zero. The guarantee they give is about input position and internal state; whether the *output* is duplicated on replay depends entirely on the sink (Checkpointing, Streaming State).
- Backpressure is a correctness mechanism disguised as a performance one. A pipeline that cannot slow its producer must either buffer without bound or drop, and both of those turn a capacity problem into a data-loss problem (Backpressure, Backpressure).
- Replay is the property that makes a log different from a queue. Because consumers hold their own offsets against retained data, a bad transformation can be fixed and re-run over the same input. That single property is why event logs became data infrastructure rather than only messaging (Retention and Replay, Offsets and Commits, Replay from the Log).
What a pipeline inherits, and from whom
Distributed Systems as a discipline asks what is provable across machines that fail independently and share no clock. This domain never re-derives those results. What it does — and what nobody else does for you — is trace which of them your pipeline is actually leaning on, and what the number on the dashboard does when the assumption fails.
The framing that makes this tractable: a pipeline's guarantee is the minimum over its hops, and consumers experience the minimum, not the maximum. A transactional warehouse fed by an at-least-once stream is an at-least-once table. A perfectly ordered source read by a job that parallelises by file is an unordered dataset. Adding a strong component downstream of a weak one changes nothing.
The matrix below is the boundary. The third column is the thing to memorise: it is not the concept, it is the specific way the concept shows up as a wrong number.
| We teach | Depth lives in | The mechanism that crosses |
|---|---|---|
| Which delivery semantic a dataset was built on | Distributed Systems: delivery semantics, failure detection, the impossibility of distinguishing a slow node from a dead one | A producer that cannot tell a timeout from a failure must retry, so at-least-once is the default everywhere. Your fact table inherits it as duplicate rows. |
| Choosing a partition key for a topic | Distributed Systems: partitioning, consistent hashing, rebalancing | Ordering exists only inside a partition, so the key decides which sequences are trustworthy and which are coincidence (Event Keys and Partition Assignment). |
| Reasoning about event time versus arrival | Distributed Systems: clocks, causality, the absence of a global now | Two producers' clocks disagree and neither is authoritative, so every window boundary is a judgement call encoded as a watermark (Watermarks). |
| Making a stream job restartable | Distributed Systems: consistent snapshots, distributed checkpointing, two-phase commit | A checkpoint aligns state with input position. The output side needs its own mechanism, which is where the guarantee usually leaks (Checkpointing). |
| Handling a consumer that cannot keep up | Distributed Systems: flow control, queueing, load shedding | Without backpressure a pipeline buffers without bound or drops. Both convert a capacity incident into a completeness incident (Backpressure). |
| Replaying to fix a bad transformation | Distributed Systems: durable logs, offsets, idempotent state machines | Replay is only correct if the transformation is deterministic and the sink is keyed. Otherwise the second run is a new kind of wrong (Replay from the Log). |
| Joining two streams | Distributed Systems: state, watermarks, out-of-order arrival | A join across independently partitioned streams is a buffering problem with a time bound, and the bound is where matches quietly stop happening (Stream Joins). |
Reading the guarantee column top to bottom
The pipeline device below is the same one used in the foundations module, applied here to a streaming path specifically. Read only the guarantees column, in order. The strongest promise is at the source, and every hop after it is either preserving or weakening what came before.
The stage that surprises people is the sink. Frameworks advertise a guarantee that is genuinely real about input consumption and internal state, and it is easy to read that as a promise about the rows in your warehouse. It is not, and the gap is closed by a keyed merge you write, not by a configuration flag (Exactly-Once: Input Consumption, State Update, Output Write).
Also notice what is absent from the whole chain, exactly as it was absent from the batch version: nothing here compares the output to the source. Completeness is not delivered by any of these stages. It is measured afterwards or it is assumed (Reconciliation).
- 1Producer
Emits an event after the business fact is durable in its own store.
guarantees At least one copy of every event it decided to send, given retries. Nothing about events it never got to send because it crashed between the commit and the emit.
fails by Dual-writing — committing to the database and publishing separately, so a crash between them loses the event permanently (The Dual Write Problem).
- 2Partitioned log
Stores events durably and serves them to independent consumer groups.
guarantees Durability, replay within retention, and a total order within one partition. No order across partitions and none for a key that changed partition.
fails by Retention expiring under a lagging consumer, which deletes unread history rather than delaying it.
- 3Consumer group
Assigns partitions to consumers and tracks progress with committed offsets.
guarantees One consumer per partition at a time, and that processing resumes from the last committed offset.
fails by Committing the offset before the effect is durable, which converts a crash into silent data loss; or after, which converts it into duplicates.
- 4Stream processor
Applies the transformation and maintains keyed state across events.
guarantees On restart, state and input position are mutually consistent, back to the last checkpoint.
fails by Unbounded state growth when a keyed store has no expiry; and by watermarks advanced by one fast partition ahead of the slow ones.
- 5Windowing
Groups events into event-time windows and emits when the watermark passes.
guarantees Each event lands in the window its event time belongs to, provided it arrives within the allowed lateness.
fails by Dropping events that exceed lateness — correctly, silently, and without any downstream signal unless you emit the count (Late Events).
- 6Sink write
Writes the result into a warehouse table or object store.
guarantees Only what the sink itself supports: a transactional sink can make the write atomic with the offset commit; an idempotent keyed merge makes a repeated write harmless. An append with no key guarantees neither.
fails by Replaying an hour of output after a restart and appending it, which doubles a metric that no test was watching.
- 7Serving table
Holds the result for query by consumers.
guarantees Query semantics and, if published atomically, that readers never see a partial state.
fails by Being read while a replay is mid-write, so a report captures a state that never existed as a whole (Atomic Publish).
The chain's guarantee is the minimum of this column. Consumers experience that minimum and have no way to see it from the table.
The five failures that look like a data bug
When a number is wrong, the first instinct is to read the transformation. On this boundary the transformation is usually innocent — it did precisely what it was told to a set of rows that had already been duplicated, reordered, truncated or dropped before it ran.
These five are worth learning as a set because their symptoms overlap and their responses do not. Duplication and a fan-out join both inflate a metric; only one of them is fixed in SQL. A short day and an outage both flatten a chart; only one of them refills when you replay.
The response column is deliberately specific. "Investigate" is not a response, and every row here has a first move that either confirms or eliminates the cause within minutes.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A producer retried after an ambiguous timeout | Revenue is high; row counts exceed the source by a small, irregular margin. | At-least-once delivery. The duplicate is a correct outcome of the transport, not a bug in it. | Count distinct business keys versus total rows in the landed table. If they diverge, add or fix the keyed merge — do not patch it in the reporting query (Upserts and Merges). |
| Topic partition count was increased | Per-entity "current state" is wrong for some entities and right for others, with no pattern in the data. | Key-to-partition mapping changed, so a key's new events are ordered against a different partition's history than its old ones. | Check the partition count change log against the first bad entity's event times. Rebuild affected keys from the source version column rather than from arrival order (Event Keys and Partition Assignment). |
| Consumer lagged past the retention window | A window of history is missing and never fills in, even after the pipeline recovers. | Retention deleted unread events. This is loss, not lateness. | Determine the exact gap from offset ranges, then backfill that range from the source system if it still has the data. If it does not, publish the gap as a known incompleteness rather than letting it be discovered (Data Incidents). |
| One partition's producer stopped while others continued | Windows close early and totals are low; the drop is proportional to one segment of the key space. | The watermark is derived from the fastest progress, so the stalled partition's events arrive past the window and are dropped as late. | Compare watermark position with per-partition maximum event time. Configure the watermark to account for idle partitions, and alert on the dropped-late counter (Watermarks). |
| A stateful job restarted from a checkpoint | A block of output is duplicated, exactly overlapping the interval between the checkpoint and the crash. | State recovery is consistent; the sink was append-only, so the re-emitted output added instead of replacing. | Make the sink keyed and idempotent so replay converges. Then re-run the affected interval and verify by count, not by inspection (Reprocessing vs Retrying). |
How to build it
Most important first.
- Write down what each hop promises before you write the transformation. Delivery, ordering scope, completeness bound, and what happens on restart. Most pipeline arguments are two people assuming different answers to those four questions (The Data Loop).
- Make every sink idempotent on a business key. A keyed merge turns at-least-once transport from a correctness problem into a non-event, and it costs a key and a merge (Upserts and Merges, Idempotent Data Pipelines).
- Choose the partition key as the entity whose ordering the business actually needs, and treat changing the partition count as a schema-level change with a migration, not a capacity slider (Event Keys and Partition Assignment).
- Aggregate on event time, and state the allowed lateness as a published property of the dataset rather than a job parameter nobody reads (Late-Arriving Data, The Freshness SLO).
- Size retention against your worst realistic recovery, not your normal lag. Retention is the width of the window in which any mistake is still fixable (Retention and Replay).
- Prefer a transactional or keyed sink over any framework setting whose name contains a guarantee. The setting is about the framework's internal state; the durable effect is yours to make idempotent (Atomic Publish).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A partitioned log guarantees durable, replayable storage and a total order within one partition. It guarantees no order across partitions, and no order for a key that has moved partition.
- A consumer group guarantees that each partition is assigned to one consumer at a time. It does not guarantee that a rebalance leaves no work half-done, which is why the commit point matters (Offsets and Commits).
- A checkpointing stream engine guarantees that on restart, operator state and input offsets are consistent with each other. Whether an output write happens once depends on the sink being transactional or the write being idempotent — the engine cannot promise that on its behalf (Exactly-Once: Input Consumption, State Update, Output Write).
- Nothing in the chain guarantees completeness. Completeness is measured by reconciliation against a source of truth, never received from transport (Reconciliation).
- Nothing guarantees that two independently partitioned streams can be joined without buffering, because the matching records may arrive arbitrarily far apart (Stream Joins).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that catches the most on this boundary is a uniqueness assertion on the business key in the serving table, paired with a count of how many duplicates the merge collapsed. The second half is what tells you the transport is duplicating rather than that it never was.
- It misses duplicates that differ in the key — a redelivered event with a fresh event id is a different row by every test you have — and it misses ordering errors entirely, because a correctly-deduplicated table can still hold the wrong version of a row.
- For ordering, the complementary check is a per-key monotonicity assertion on the source's version or log position. It catches partition-count changes and out-of-order merges, and it misses anything where the source itself assigned versions unreliably.
- A log-based path removes the schedule from the freshness equation — the lower bound becomes producer batching plus consumer lag rather than "the next run".
- Event-time windowing deliberately *adds* latency: a window cannot emit until the watermark passes its end, and the allowed lateness is exactly the delay you are buying completeness with (Windows).
- Consumer lag is the honest freshness signal for a streaming path, and it is the one number a consumer of the data should be able to see (Freshness Monitoring).
- Increasing partition count changes the key-to-partition mapping and therefore breaks per-key ordering across the change. Treat it as a breaking change with a documented cutover, not as scaling (Topics and Partitions).
- A producer schema change propagates to every consumer group independently and at whatever speed each deploys, so producers and consumers are always running different versions of the contract for a while (Schema Registry, Backward Compatibility).
- Changing a window definition or an allowed-lateness setting changes historical results if the job is ever replayed, which makes those settings part of the dataset's definition rather than its configuration.
- Replay is the primary recovery mechanism and its range is bounded by retention. Reset the consumer group to an offset or timestamp, re-run the transformation, and write to a location consumers are not reading yet (Replay from the Log, Reprocessing vs Retrying).
- Replay is only safe if the transformation is deterministic and the sink is idempotent. A transformation that calls
now(), reads a mutable lookup table, or appends without a key will produce a different and worse result on the second run (Idempotent Data Pipelines). - State-bearing jobs recover from checkpoints, and a checkpoint older than the retention window is not recoverable at all — the input it would need has been deleted (Checkpointing).
What can go wrong
- Duplicates from at-least-once delivery, arriving as an inflated metric that people question far less readily than a deflated one.
- Ordering lost by a partition-count change, which produces a wrong "current state" per key and no error anywhere.
- Retention expiring under a lagging consumer: the data is not late, it is gone.
- A watermark advanced by a single fast partition, closing windows before the slow partitions' events arrived.
- The mitigation failing: a deduplication step keyed on an event id that the producer regenerates on retry, which passes every test and removes nothing.
- Unbounded state in a stateful job, because a keyed store was never given a retention policy and the key space is a customer id (Streaming State).
- "The broker guarantees exactly-once, so my pipeline does." A broker can offer transactional writes and offset commits for consumption and state within its own boundary; the moment your output goes to a warehouse, an object store or an API, that guarantee stops at the boundary and your idempotent write is what carries it (Exactly-Once: Input Consumption, State Update, Output Write).
- "Events are ordered." Events are ordered within a partition. If your reasoning involves two keys, two topics, or a key that has ever moved partition, you do not have the order you think you have.
- "Streaming is more modern, so it is better." They answer different questions. Compare them on freshness need, complexity, failure handling, cost shape and who is on call, and a large share of streaming projects come out on the batch side (Batch vs Streaming Ingestion).
- "Increasing partitions is just scaling." It rewrites the key-to-partition mapping and breaks per-key ordering across the boundary. It is a migration.
- "We are not doing distributed systems, we are doing data." A pipeline with a broker, a stream processor and a warehouse is a distributed system whose failures happen to look like wrong numbers.
Operating it
- Consumer lag per partition, not per topic. The aggregate hides the one partition that is stuck, which is the usual shape of the problem (The Backlog Arithmetic: Four Levers and a Drain Time).
- Watermark position versus wall clock per job, and the count of records dropped for exceeding allowed lateness — a number that should never be zero and never be large (Pipeline Metrics).
- Duplicates collapsed by the merge, per run, as a trend. A step change means something in the transport path changed (Volume Anomalies).
- Checkpoint duration and interval; a checkpoint that keeps growing is state that is never being released (Checkpointing).
- At 10x event volume the constraint is usually one hot key rather than the total, because ordering pins a key to a partition and a partition to a consumer (Data Skew, Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- At 100x, state size becomes the binding constraint on stateful jobs long before throughput does, and every keyed store needs an explicit expiry.
- Consumer count scales fan-out on the log rather than the pipeline: a log with many independent consumer groups multiplies read bandwidth but leaves each group's correctness independent, which is the property that made this architecture attractive in the first place (The Event-Driven Data Platform).
- Retention is the dominant storage driver on this boundary and it is bought deliberately: it is the width of your recovery window, and shortening it to save storage shortens how long a mistake stays fixable (Retention and Replay).
- Partition count drives per-partition overhead on the broker and file-count pressure on any sink that writes one file per partition per interval (File Size and the Small-Files Problem).
- Stateful streaming costs continuously held compute and memory whether or not anything is happening, which is the structural difference from a batch job that costs only while it runs (Cost vs Freshness).
- Every guarantee is bought with latency, state or coordination. Event-time correctness costs the watermark delay; effectively-once output costs a keyed or transactional sink; ordered processing costs the ability to parallelise within a key.
- A streaming path removes the schedule and adds a permanently running system with its own state, its own recovery procedure and its own on-call. It is not the batch pipeline with a smaller interval (Batch vs Streaming Ingestion).
- Long retention makes mistakes fixable and makes deletion obligations harder, because a replayable log is a copy of personal data with its own lifetime (Data Retention).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat guarantees are inherited from the weakest hop, that ordering is scoped to a partition, and that event time diverges from processing time hold for every distributed data path regardless of product.
- BROKER-SPECIFICKafka orders per partition and lets a consumer group hold its own offsets over retained data, so replay is a reset; Kinesis orders per shard with a bounded retention; Pub/Sub gives no ordering at all unless an ordering key is set, and its replay model is a subscription seek rather than an offset. A design that assumes offsets does not port to a system that has none.
- ENGINE-SPECIFICFlink checkpoints operator state and input positions together and can drive a two-phase commit to a transactional sink; Spark Structured Streaming uses a write-ahead log plus idempotent sinks with a different recovery shape. What both share is that the output guarantee is the sink's, not the engine's.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems does not exist in Engineer Atlas yet. When it lands it owns the depth on: consensus and leader election, failure detection and the impossibility of distinguishing slow from dead, consistency models from linearizable to eventual, logical clocks and causality, distributed snapshots, and two-phase and three-phase commit. Every one of those is a promise this domain consumes and never proves.
- — Specifically inherited by a data pipeline and taught nowhere else here: why at-least-once is the default rather than a defect (an ambiguous timeout leaves no correct choice but to retry), why total ordering across partitions is not offered rather than merely unimplemented, and why a watermark is a heuristic about a system that has no global clock rather than a measurement.
- — The reverse direction matters too: the durable partitioned log, checkpointing, and idempotent state machines are distributed-systems constructions that this domain adopted wholesale. A learner who reads the Distributed Systems domain first will recognise every mechanism in the streaming module.