Offsets and Commits

• An offset is a position in a partition, and committing it stores that integer for the consumer group so a restart or a reassignment resumes from there. It is the entire state a consumer keeps — but it is a separate action from the processing, and the two cannot be made atomic by ordering alone. The only question is which one you do first, and that choice is the delivery semantic (The Event Log).
• Commit first, then process gives at-most-once: a crash in between loses the records, because the group has already claimed to be past them. Nothing is ever duplicated and records can silently disappear.
• Process first, then commit gives at-least-once: a crash in between reprocesses the records, because the group has not yet claimed to be past them. Nothing is ever lost and records can be duplicated.
• There is no third ordering. The only way to remove the gap is to make the offset commit and the output write one atomic operation, which requires a sink that can participate in a transaction with the offset store — a narrow and specific condition, not a broker feature you enable (Transactions and ACID).
• The practical alternative, and the one most data platforms actually use, is at-least-once delivery with an idempotent sink: write keyed on a stable business identifier so a redelivered record overwrites itself rather than adding a row. The effect is once; the delivery is not (Upserts and Merges).
• Automatic commit is process-first ordering with an unpredictable boundary: the commit fires on a timer, not at a point you chose, so the amount of work at risk varies and the ordering guarantee is weaker than it appears.

1# The shape common to log-client libraries: poll for records, do work,
2# commit a position. No library-specific API is assumed beyond that.
3 
4# (A) AT-MOST-ONCE — commit first.
5for batch in consumer.poll():
6    consumer.commit()          # claim "done" before it is true
7    write_to_warehouse(batch)  # crash here: these records are gone
8 
9# (B) AT-LEAST-ONCE — commit after a DURABLE write.
10for batch in consumer.poll():
11    write_to_warehouse(batch)  # must be durably acknowledged, not buffered
12    consumer.commit()          # crash before this: batch is reprocessed
13 
14# (C) EFFECTIVELY-ONCE EFFECTS — (B) plus an idempotent sink.
15for batch in consumer.poll():
16    upsert_to_warehouse(batch, key="event_id")  # re-writing is harmless
17    consumer.commit()
18 
19# There is no (D) that removes the gap by reordering. Removing it requires
20# the offset commit and the write to be ONE transaction, which requires a
21# sink that can participate in it.

partition 3

  offsets:   ... 4808  4809  4810  4811  4812  4813  4814 ...
                             ^                       ^
                             committed = 4810        position = 4814
                             |                       |
                             +--- the at-risk window -+
                             records 4810..4813: read, maybe processed,
                             not yet claimed as done

  crash here, commit-AFTER-process  ->  4810..4813 read again
                                        (at-least-once; duplicates)
  crash here, commit-BEFORE-process ->  4810..4813 never processed
                                        (at-most-once; silent loss)

  The window cannot be made zero by ordering.
  It can only be made ATOMIC, and only if the sink can join
  the same transaction as the offset store.

• Choose at-least-once and make the sink idempotent: for a data platform this is almost always right, because reprocessing a record is recoverable and losing one is not. Turn automatic commit off and commit explicitly after the write has been durably acknowledged, so the boundary is one you chose and can reason about (Checkpointing).
• Give every record a stable identity that survives redelivery — an event id, or source table plus primary key plus log position — and key the sink write on it. Without that, idempotency is not implementable and at-least-once becomes duplication (Deduplication).
• Prefer an upsert or an atomic partition swap over an append. Appending is what turns redelivery into duplicate rows, and appending is the default in most file-based sinks (Atomic Publish).
• Size the batch between commits deliberately. It is the amount of work reprocessed after any failure, and it trades commit overhead against redo cost — a real knob with a real cost on both ends.
• Never claim exactly-once without naming which of input consumption, state update and output write it covers, and what assumption buys it. The claim is meaningless without that scoping and misleads every consumer who hears it (Exactly-Once: Input Consumption, State Update, Output Write).
• Handle the poison record explicitly: catch, publish to a quarantine topic, commit past. Without it a single unparseable record stops the partition permanently (A Dead-Letter Queue Is a Workflow, Not a Bin).

• Commit-before-process guarantees no duplicates and does not guarantee no loss. This is at-most-once and is almost never the right choice for data.
• Commit-after-process guarantees no loss and does not guarantee no duplicates. This is at-least-once and is the correct default for a data platform.
• An idempotent sink keyed on a stable identifier gives effectively-once effects on the output, while delivery remains at-least-once. The guarantee lives in the sink, not in the broker.
• A transaction spanning the offset commit and the output write gives exactly-once for that specific pair, and only when the sink can join that transaction. It says nothing about any other sink the pipeline writes to.
• No configuration gives end-to-end exactly-once delivery across a broker and an external warehouse or object store. Anyone who says otherwise has not said which of the three stages they mean (Exactly-Once: Input Consumption, State Update, Output Write).
• A committed offset guarantees that a consumer moved past a record. It guarantees nothing whatsoever about whether the record was processed correctly, or at all (The Pipeline Succeeded. The Data Is Wrong.).