Exactly-Once: Input Consumption, State Update, Output Write
There is no single exactly-once guarantee — there are three separate questions, one about input consumption, one about state update and one about output write, and each is bought by a different, nameable assumption.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A streaming platform advertises that every event is processed once and only once. Which of the three things that could mean is being promised, and what has to be true for it to hold?
The revenue metric that must not double when a job restarts, the alerting rule that must not fire twice for one incident, and the payment service that must not be told twice to charge a card. The first two want a correct number; the third wants a correct effect, and only one mechanism in this lesson can give it that (Idempotent Data Pipelines).
The unit here is the effect, not the record. One input record can be read many times, cause many attempted state updates and produce many attempted writes, and still be correct — provided the observable result is the same as if it had been read, updated and written once. Reasoning about records rather than effects is what makes this topic confusing.
Enable the setting the engine calls exactly-once processing, deploy, and stop thinking about duplicates. This is not lazy — it is what the configuration page implies, the setting is real, and for a job whose entire world is one broker it comes remarkably close to being true.
The job writes its results to a warehouse table over JDBC. The engine's guarantee covers its own state and its own source offsets; the external write is outside the commit, so a restart between the write and the checkpoint replays the write and the table gains a duplicate batch (Atomic Publish).
- The job writes its results to a warehouse table over JDBC. The engine's guarantee covers its own state and its own source offsets; the external write is outside the commit, so a restart between the write and the checkpoint replays the write and the table gains a duplicate batch (Atomic Publish).
- The transformation calls
now()to stamp a processed-at column. On replay the same input produces a different row, so the write is repeated rather than repeating — the mechanism assumed determinism and the code quietly removed it (Deterministic Replay: Making the Schedule Reproducible). - A file source is used instead of a log. The engine restarts, asks the source to replay from a position, and the source has no positions — a directory listing is not an offset. The guarantee had a precondition nobody stated (Replay from the Log).
- The sink is a REST API belonging to another team. There is no transaction to join and no key it deduplicates on, so a retry creates a second record on their side and one on yours, and only they can see the divergence (Webhook Idempotency).
- Everything works and the downstream consumer reads uncommitted output because it was configured to read as fast as possible. It observes records that a later abort rolls back, and no amount of correctness upstream repairs a consumer that read the wrong thing (Offsets and Commits).
- The upstream producer retried before any of this began. Two records describing one business event entered the log with different record identities, and every mechanism in this lesson faithfully processes both of them once (Deduplication).
What is actually happening
- Split the phrase into three claims before evaluating any of them. Input consumption: is each input record delivered to the operator once? State update: does each input record affect the operator's accumulated state once? Output write: does each result appear in the destination once? They are independent, they fail separately, and almost every argument about this topic is two people answering different ones (Stateful Stream Processing).
- Input consumption, at the transport level, is at-least-once and cannot be otherwise: the receiver cannot distinguish a lost record from a lost acknowledgement, so it must be willing to be sent the record again. No configuration changes that; what configuration changes is what happens to the *duplicate*.
- State update is made to look like a single application by atomically committing operator state together with the input position. On restart, state is restored from the snapshot and the input is replayed from exactly the position that snapshot covered — so the records processed after it are processed again, against state that has been rolled back to before they were. The duplicate work is real; the duplicate effect is not (Checkpointing).
- That rollback-and-replay only produces the same state if processing is deterministic. Wall-clock reads, random numbers, iteration over an unordered collection, and any lookup against a system that has changed since — each of them turns replay into a different computation whose result is merely plausible (Nondeterminism: Same Input, Different Output).
- Output write is the hard one, because a snapshot of your own state cannot roll back somebody else's system. There are exactly three mechanisms: a transactional sink that commits with you and is aborted with you; an idempotent write keyed by something derived from the input, so repeating it overwrites rather than accumulates; or downstream deduplication over a bounded window, which is idempotency implemented by the reader instead of the writer (Upserts and Merges).
- A transactional sink imposes a visibility rule on the consumer as well as the producer. Output becomes visible only at commit, so end-to-end latency is tied to the commit interval — and a consumer that does not read committed-only sees the uncommitted records anyway (Atomic Publish).
- Nothing in any of this reaches an effect that leaves the system — an email, a card charge, a message to a partner. Those are covered only by a key the *receiver* honours, which makes the guarantee theirs rather than yours (Idempotency Keys: The Mechanism).
- The event-time machinery this rests on — window assignment by event time, expiry by watermark — is modelled in
src/de/sim/stream.tsand pinned byscripts/de-sim.test.ts. That model is what makes replay reproducible in the first place: the same events replayed produce the same window membership and therefore the same output to be written once (Watermarks).
Three questions wearing one name
The phrase is a compression of three separate engineering questions, and the compression is where the trouble starts. A vendor answering the second one and an engineer hearing the third one will have a completely satisfactory conversation and ship a duplicated revenue table.
Read the guarantees column below one row at a time and notice that each row is bought by a different assumption. The first is bought by a replayable source. The second is bought by an atomic commit of state with position, plus determinism in the code between them. The third is bought by the destination, and the destination is usually somebody else's system.
The failsBy column matters more than the other two. Each stage fails in a way that produces no error, and in every case the observable symptom is a number that is slightly too large — which is the direction people are least likely to question.
- 1Input consumption
Reads records from the source at a position the job can return to.
guarantees At-least-once delivery to the operator, and the ability to rewind to a committed position. Never that a record is read only once — after a restart, records past the last commit are deliberately read again.
fails by A source with no stable positions. The engine asks to rewind, the source cannot, and the span between the snapshot and the crash is gone with no error raised (Replay from the Log).
- 2State update
Applies each record to accumulated operator state, and periodically commits that state together with the input position.
guarantees Effectively-once state: after recovery, state equals what one application of each record would have produced — provided the code between input and state is deterministic.
fails by Any non-determinism in the transformation. A wall-clock read or an unordered iteration makes replay a different computation, and the resulting state is plausible rather than correct (Nondeterminism: Same Input, Different Output).
- 3Output write
Publishes results to a destination the engine does not own.
guarantees Nothing by default. With a transactional sink, atomicity of the batch with the state commit; with an idempotent write, that repetition overwrites; with downstream deduplication, that repeats inside the window are discarded.
fails by An append-only sink outside the commit. Every restart re-appends the replayed span, and the duplicates are indistinguishable from real volume (Duplicate Rows).
- 4External side effect
Calls a system outside the platform — a payment, an email, a partner API.
guarantees Whatever the receiver promises, and nothing more. No commit protocol on your side can withdraw a message that has already left it.
fails by A retry without a key the receiver honours, producing a second real-world action that only a compensating action can undo (Idempotency Keys: The Mechanism).
A platform is only as strong as the weakest row it actually implements. Enabling the engine setting answers rows one and two; rows three and four are design decisions that no configuration page contains.
The idempotent write is the general answer
Given a choice between a cross-system transaction and an idempotent write, prefer the write. The transaction protects one specific path — the engine's own commit — and protects nothing when a human re-runs the job, when a backfill covers the same range, or when the sink is replaced. The idempotent write protects all of them, because it is a property of the destination rather than of the run.
It rests on one requirement that is easy to state and easy to violate: the key must be a pure function of the input. Window boundary plus grouping key. Source partition plus offset. A business identifier that the producer assigned. Anything that varies between attempts — an attempt number, a generated identifier, a hostname, a timestamp taken at write time — converts the merge into an insert and does so without changing a single line of the surrounding logic.
The second requirement is that the merge replaces rather than accumulates. The two forms differ by a handful of characters and are indistinguishable in a code review that is looking at the join condition instead of the assignment. An accumulating merge is a correct-looking counter that a replay increments a second time, and the pipeline is green throughout.
Enable the engine's guarantee, write results to the destination with an append, and treat duplicates as an engine concern. Restarts are rare, the setting is on, and the table has never obviously been wrong.
Derive a key from the input, merge with a replacing assignment, and assert uniqueness at that grain as a data test on the destination. The engine setting is then one of two independent protections rather than the only one.
The engine's commit covers its own state and its own source positions. It cannot roll back a write to a system it does not participate in, so an append-only destination is unprotected no matter what the engine is configured to do — and the failure surfaces only after a restart, which is exactly when nobody is examining row counts.
1-- The key is (window_start, user_id): both come from the input, so the same2-- input produces the same key on every attempt, replay and backfill.3MERGE INTO fct_window_totals AS t4USING (5 SELECT window_start, user_id, sum(amount_minor) AS amount_minor6 FROM staged_results7 WHERE window_start = :window_start8 GROUP BY window_start, user_id9) AS s10ON t.window_start = s.window_start11AND t.user_id = s.user_id12-- REPLACE. This is the line that makes the write idempotent.13WHEN MATCHED THEN UPDATE SET amount_minor = s.amount_minor14-- ... = t.amount_minor + s.amount_minor <-- accumulates; a replay doubles it15WHEN NOT MATCHED THEN INSERT (window_start, user_id, amount_minor)16VALUES (s.window_start, s.user_id, s.amount_minor);Two things to notice. The key contains nothing about the run — no attempt id, no hostname, no write timestamp — and the matched branch assigns rather than adds. Change either and the statement still parses, still runs, and stops being idempotent.
Where the claim actually dies
Every incident in this area has the same shape: a mechanism that was correct in its own terms met an assumption nobody had written down. The table below is those meetings. None of the rows involves a bug in an engine — each is a correct mechanism applied where its precondition did not hold.
Read the cause column as a list of preconditions. A replayable source. Deterministic code. A participating or idempotent sink. A consumer configured to match. A key derived from the input. A receiver that honours idempotency. Six sentences, and a platform that can state all six is in a different position from one relying on a checkbox.
The response column is deliberately dull. There is no clever recovery from any of these; the work is done in advance, and the only genuinely irreversible entry is the last one, where an effect has already left the building.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Job restarts; sink is an append-only table. | A block of duplicate rows covering the replayed span, and a metric that is high by an amount nobody can name. | The engine commit covers its state and positions. The external write was never in it. | Convert the write to a merge on a key derived from the input, and assert uniqueness at that grain (Upserts and Merges). |
Transformation stamps processed_at = now(). | Replayed rows overwrite correctly but with different values, so history changes quietly after every restart. | Replay was assumed deterministic and the code made it a function of when it ran. | Derive every value from the input. If a processing timestamp is genuinely needed, take it from the checkpoint or the event, not the clock (Deterministic Replay: Making the Schedule Reproducible). |
| Source is a directory of files that a mover deletes. | After a restart, a gap in the output that no error accompanies. | The engine asked to rewind to a committed position; a directory listing has no positions. | Land raw arrivals immutably and consume from a log or from an immutable landing zone with stable positions (The Raw Landing Zone). |
| Consumer reads uncommitted output from a transactional sink. | The consumer occasionally reports records that later cease to exist upstream. | Producer-side atomicity has a consumer-side precondition that was left at its default. | Set the consumer to read committed-only, and treat that setting as part of the contract for the topic (Data Contracts). |
| Output key includes the attempt number. | Two rows per window per key after a retry, both looking entirely legitimate. | The merge condition never matches, so every attempt inserts. | Key on the input alone. Review keys the way you would review a primary key, because that is what it is (Surrogate Keys). |
| Backfill re-runs a range the streaming job already published. | Doubled totals for the backfilled range only, discovered a week later during a month-end close. | The streaming write was protected by the engine's commit; the batch write was a separate code path with a separate author. | Make both paths write through the same idempotent merge, so protection is a property of the destination rather than of which job wrote (Planning a Backfill). |
| Retry of an external call with no idempotency key. | The partner has two records; you have one; reconciliation blames them. | The receiver deduplicates on a key it was never sent. | Send a key derived from the input and confirm it is honoured. Where it is not, do not retry automatically — queue for a compensating action (Webhook Idempotency). |
Reading a guarantee somebody else wrote
The practical skill this lesson is for is reading a claim in a document and converting it into the three questions before deciding what it is worth. The decision device below is the interrogation, and its options are the mechanisms you can actually choose between once the marketing has been removed.
There is no winner here and the criteria genuinely differ by destination. A key-value sink makes idempotent writes nearly free; a columnar table makes them a file rewrite. A broker with transactions makes the second option available inside its own boundary and nowhere else. A partner API makes the whole question theirs.
What does not differ is the discipline: name which of the three questions is being answered, name the assumption that buys it, and then write a data test that would notice if the assumption stopped holding. The test is not redundancy — it is the only part of the arrangement that observes the property rather than believing it (Data Tests).
The state is handled by the engine. What makes the write to the destination safe to repeat?
when The destination supports transactions, and the batch must appear all-or-nothing — a partial window being visible would be worse than a late one.
cost Visibility is tied to the commit interval, every consumer must read committed-only, and the coordination is paid on every commit. Cross-system two-phase commit is also the hardest part of any engine to operate (Atomic Publish).
when The destination supports merge semantics. This is the default answer for warehouse and lakehouse sinks, and the only one that also protects manual re-runs and backfills.
cost A merge rewrites files on columnar destinations, so scattered keys are materially more expensive than clustered ones, and the correctness depends on a key discipline that nothing enforces (Upserts and Merges).
when The unit of output is a partition and the job can always recompute it in full — the common shape for windowed aggregates.
cost The whole partition is rewritten even when one key changed, and any late record arriving after the overwrite requires the partition to be recomputed again (Late-Arriving Data).
when The sink is out of your control and appends, and repetitions are known to arrive close together.
cost A state store of seen keys sized by the window, and correctness that stops at the window boundary — a repetition arriving later is indistinguishable from a new record (Deduplication).
when The consumer can tolerate them, or the volume is low enough that a reconciliation job can flag them for a human.
cost Every downstream query must be duplicate-aware, and that obligation spreads to consumers who do not know they have it. Honest, and it ages badly (Reconciliation).
Transaction support, isolation-level defaults, sink connector semantics and state-backend compatibility across versions all change between releases of every engine and broker named in this lesson, and connectors frequently document a guarantee the underlying destination cannot support. Verify against current documentation for your exact versions, and test it by restarting the job under load rather than by reading the page.
How to build it
Most important first.
- Write the claim down as three lines before you accept it. "Input replayed from committed offsets; state restored with those offsets atomically; output written idempotently on
(window_start, key)." If any line cannot be completed, that is the line that will page you (Data Contracts). - Prefer an idempotent write on a deterministic key to a transactional sink wherever the destination supports a merge. It survives sink restarts, job restarts, manual re-runs and backfills, none of which a transaction spanning two systems survives well (Upserts and Merges).
- Derive the output key from the input — window boundary plus grouping key, or source offset, or a business identifier — never from the run, the attempt, the hostname or the clock. A key containing anything that changes between attempts converts an idempotent write into an append (Idempotent Data Pipelines).
- Make the merge replace, not accumulate.
SET total = new.totalis idempotent;SET total = old.total + new.totalis a counter that a replay increments a second time, and the two differ by one character in review (Upserts and Merges). - Ban non-determinism in transformation code and enforce it in review: no wall-clock reads, no random identifiers, no unordered iteration, no lookups against mutable state that is not itself part of the checkpoint (Deterministic Replay: Making the Schedule Reproducible).
- Use a replayable source. A log with stable offsets can be rewound; a socket, a queue that acknowledges destructively, or a directory of files being moved cannot, and the strongest engine configuration cannot reconstruct data the source no longer has (Retention and Replay).
- Configure the consumer to match the producer. A transactional sink and a consumer reading uncommitted records produce a system that is careful at one end and wrong at the other (Offsets and Commits).
- For external side effects, send an idempotency key derived from the input and confirm in writing that the receiver honours it. Where they do not, the effect must be made safe some other way — a reconciliation job, a bounded retry, or accepting duplicates and detecting them (Reconciliation).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- What a checkpointing engine genuinely promises is effectively-once state: after any restart, the operator state equals what it would have been had each input record been applied once. It does not promise the record was read once, and reading it again is how the promise is kept (Checkpointing).
- Output write is guaranteed only to the extent the sink participates. A transactional sink promises atomicity of the batch with the state commit; an idempotent sink promises that repetition is harmless; a plain append sink promises nothing at all, whatever the engine setting says.
- Ordering is unchanged by any of this. Per-partition ordering survives; global ordering was never available and no commit protocol creates it (Topics and Partitions).
- Completeness is a separate question again. A job can be correct in all three senses and still be missing every record its watermark classified as late (Late Events).
- What is explicitly not guaranteed: anything about duplicates that entered upstream. Two records for one business event are two distinct events to the whole machinery, and only a business-key deduplication sees them as one (Deduplication).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Assert uniqueness at the output grain — one row per window per key, or one row per business key — as a data test on the destination rather than a property assumed from configuration. It is the only check that observes the guarantee instead of trusting it (Data Tests).
- Reconcile the streaming output against a batch recomputation of the same range from retained raw data. A duplicate that both paths share is upstream; one that only the streaming path has is a write that was not idempotent (Reconciliation).
- Both miss the duplicate that entered before ingestion, because it is not a duplicate to anything in the pipeline — two records with different identities describing one real event reconcile perfectly and are still wrong (What a CDC Event Contains).
- They also miss non-determinism that produces a *different* row rather than a second one. A replay that stamps a new processed-at value overwrites idempotently and quietly changes history (Semantic Changes).
- A transactional sink couples visibility to the commit interval: results exist but cannot be read until the transaction commits, so shortening the interval improves freshness and increases commit overhead in the same move (Checkpointing).
- An idempotent write has no such coupling — the row is visible when written and a later attempt overwrites it. Freshness is unaffected, at the price of a consumer that may momentarily read a value about to be replaced.
- Downstream deduplication adds the deduplication window to end-to-end latency in the worst case, because a record cannot be declared new until the window it is checked against is complete.
- Recovery has a freshness shape of its own: after a restart, the replayed span is reprocessed before new records are touched, so the output stalls for as long as the replay takes and then catches up in a burst (The Backlog Arithmetic: Four Levers and a Drain Time).
- Changing the job's logic and restarting from an old checkpoint replays records through new code against state built by the old code. The state is not wrong in a way any type check finds — it is a mixture (Reprocessing vs Retrying).
- Adding an operator changes the state layout, so an existing snapshot may no longer be restorable. Whether it is depends on whether operators are identified by a stable declared identifier or by position in the graph, which is a property of the engine and not of your code (Checkpointing).
- Changing the output key changes what "the same result" means. Rows written under the old key are not overwritten by rows written under the new one, so the destination holds both generations and every aggregate over it doubles (Upserts and Merges).
- Switching a sink from transactional to idempotent — or the reverse — is a change to the visibility semantics consumers experience, with no schema change to notice it by (Data Contracts).
- The normal recovery is the mechanism itself: restore state from the last snapshot, rewind the source to the position that snapshot covered, and reprocess. The reprocessed span is duplicated work by construction and duplicate-free in effect (Checkpointing).
- When state is unrecoverable, recovery becomes a rebuild: replay from the log with a fresh state backend, or recompute the affected range in batch from retained raw data and merge the result over the top (Replay from the Log).
- Any recovery that involves a manual re-run relies on the same idempotent write the automatic path uses. A platform whose correctness depends only on engine configuration cannot be repaired by a human without creating duplicates (Planning a Backfill).
- Recovery of an external side effect is not possible in this layer. It is a compensating action — a refund, a correction, an apology — and designing for that is cheaper than discovering it (Saga Pattern).
What can go wrong
- The guarantee is enabled and the sink is a plain append. Every restart appends the replayed span again, and the duplicates look like a busy afternoon (Duplicate Rows).
- Non-deterministic transformation code, so replay produces plausible-but-different results and the two runs never disagree loudly enough to be noticed (Nondeterminism: Same Input, Different Output).
- A non-replayable source, so the engine restores state to a position the source cannot return to and the records in between are simply gone (Missing Rows).
- A consumer reading uncommitted output from a transactional sink, observing records that are later aborted, with the correctness failure entirely on the reading side.
- The output key contains an attempt number, a hostname or a timestamp, so the merge inserts where it was meant to overwrite (Idempotent Data Pipelines).
- The mitigation failing: shortening the commit interval to improve freshness, which multiplies commit overhead and small files in the destination until the sink becomes the bottleneck (File Size and the Small-Files Problem).
- Upstream duplication that the whole mechanism is blind to, because it operates on record identity and the duplication happened before record identity was assigned (CDC Failure Modes and the Retention Deadline).
- "Kafka guarantees exactly-once" is the sentence this lesson exists to refute, because Kafka transactions let a producer atomically commit output records and consumed offsets within Kafka, and consumers must read committed-only to benefit; a write to a warehouse, an object store or a partner API is outside that transaction and gets no protection from it (Kafka as a Log, Not a Queue).
- "We enabled exactly-once processing, so duplicates are impossible." Duplicates upstream of ingestion, duplicates from a non-idempotent external sink, and duplicates created by a manual re-run are all untouched by the setting. It covers state, and it covers the sink only if the sink participates (Atomic Publish).
- "Exactly-once processing means each record is read once." It means the opposite in practice: records after the last checkpoint are deliberately read again, because replaying them against rolled-back state is the mechanism by which the effect happens once (Checkpointing).
- "At-least-once plus a unique key is a weaker design." It is often the stronger one. Idempotent writes on a deterministic key survive backfills, manual re-runs and sink replacements, none of which a cross-system transaction handles gracefully (Idempotent Data Pipelines).
- "The engine handles it, so the transformation code does not matter." The guarantee is conditional on deterministic replay. One call to the wall clock inside a transformation invalidates it, silently, with no error and no failing test (Deterministic Replay: Making the Schedule Reproducible).
Operating it
- A uniqueness violation count at the output grain, published as a metric rather than only as a failing test. It is the direct observation of the property everything else is indirect evidence for (Quality Alerting).
- Restart count and, for each restart, the size of the replayed span. Duplicates in the destination should correlate with nothing; if they correlate with restarts, the write is not idempotent (Pipeline Metrics).
- Commit duration and commit failure rate for the sink transaction. A creeping commit duration is the earliest warning that the interval and the state size have drifted out of balance (Checkpointing).
- Row counts of the streaming output against the batch recomputation of the same range, tracked as a series rather than checked once (Reconciliation).
- For external calls, the receiver's duplicate-rejection count, which is a signal you must ask them for and will not otherwise have (Webhook Idempotency).
- At 10x throughput the state grows and the snapshot at each commit grows with it, so the commit interval that was comfortable becomes the thing that limits throughput (Checkpointing).
- At 10x parallelism a transactional commit coordinates across more participants, so the slowest participant sets the commit duration for all of them and a single degraded task stalls every output (Straggler Tasks).
- At 100x key cardinality, downstream deduplication stops being viable — the seen-key store approaches the size of the data — and idempotent writes keyed on the business key become the only affordable mechanism (Partition Cardinality).
- Consumer count changes nothing about the guarantee and everything about the blast radius: one non-idempotent sink among twenty consumers is one wrong dataset, and it will be discovered by whoever depends on it rather than by you (Impact Analysis).
- Snapshotting state at every commit costs write bandwidth and storage proportional to state size and inversely proportional to the interval. Halving the interval roughly doubles the snapshot work for the same throughput (Streaming State).
- A transactional sink costs a coordination round per commit plus the retained uncommitted data, and it costs small files in file-based destinations because each commit is a boundary (File Compaction).
- An idempotent merge costs a lookup or a rewrite per key in the destination — cheap in a key-value store, materially more expensive in a columnar table where a merge rewrites files (Upserts and Merges).
- Downstream deduplication costs a state store of seen keys sized by the window, which is a second copy of the identity space and often larger than people expect (Deduplication).
- A transactional sink buys atomic visibility of a batch and costs latency tied to the commit interval, coordination overhead per commit, a hard requirement that consumers read committed-only, and a sink that must support transactions at all.
- An idempotent write buys robustness against every kind of repetition — restarts, manual re-runs, backfills — and costs a destination that supports merge semantics plus the discipline of keeping the key derived purely from the input.
- Downstream deduplication buys independence from both the sink and the engine and costs a bounded window, which means it is correct for repetitions inside the window and blind to those outside it.
- All three cost the same thing in the end: the mechanism is invisible in the output, so a consumer cannot tell which one is protecting them, and the only way to keep it honest is a test on the data (Data Tests).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe decomposition into input consumption, state update and output write is a property of the problem: an operator cannot roll back a system it does not own, so the third question always requires a separate mechanism regardless of which engine is asking it.
- ENGINE-SPECIFICFlink commits operator state and source offsets together via a barrier-aligned snapshot and supports two-phase-commit sinks; Kafka Streams uses Kafka transactions so its state changelog, its output topic and its offsets commit atomically as long as everything involved is Kafka; Spark Structured Streaming pairs an offset log with a commit log and requires the sink to be idempotent by design rather than transactional. The three questions are the same; which of them the engine answers for you is not.
- BROKER-SPECIFICKafka offers producer idempotence per partition plus transactions spanning topics and offsets, with a consumer-side isolation level that decides whether uncommitted records are visible; Kinesis and Pub/Sub offer neither a producer transaction nor a committed-read isolation level, so on those the third question must be answered by an idempotent sink or downstream deduplication.
- SIMPLIFIEDThe three questions are treated as independent here. In a job with several stateful operators each internal hand-off is itself an output write, so a long pipeline asks the third question once per boundary and the weakest boundary decides the whole chain.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the result underneath all of this: a sender cannot distinguish a lost message from a lost acknowledgement, which is why input consumption is at-least-once by necessity rather than by implementation choice, and why the interesting engineering is entirely about what happens to the repeat.
- — Distributed Systems also owns two-phase commit and its failure modes — the blocked participant, the coordinator that dies holding the decision — which is what a transactional sink actually is once the connector name is removed.
- — DevOps / Production Engineering owns the restart itself: a rolling deploy of a streaming job is a deliberate restart, so every guarantee in this lesson is exercised on every release, and the deployment strategy decides how often the replay path is taken.