The question this answers
When should my consumer commit its position, and why can I not have both no loss and no duplicates?
Committing after processing gives at-least-once: no record is skipped, and records between the last commit and a crash are reprocessed. Committing before processing gives at-most-once: no record is processed twice, and records in flight at a crash are never processed. A stronger guarantee requires the offset and the output to be committed atomically, which is only possible when they live in the same transactional store.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The consumer knows which offsets it has fetched and which it has finished processing. It does not know whether its last commit was durably recorded — a commit is a remote call with its own A Timeout Tells You Nothing About Whether It Happened — nor whether it still owns the partition it is committing for. On restart it knows only the last offset the store admits to having, which may be behind what it actually processed.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The two orderings, and why the third does not exist
The consumer does two durable things: it produces an effect (writes a row, calls an API, emits a record), and it records its position. These are writes to two different systems. There is no transaction spanning them, so one must happen first, and whichever is second may never happen.
Commit first, then process. If the process dies between the two, the offset says the record is done and it never was. Records are lost, permanently and silently, and the group reports zero lag. Nothing anywhere reports the gap.
Process first, then commit. If the process dies between the two, the offset still points at the older position and the records are processed again after restart. Duplicates, but no gaps. This is the correct default for nearly everything, and it is why every mature stream consumer is built on idempotence.
The instinct to look for a third option is right, and the search terminates quickly: the only way to make "the effect happened" and "the position advanced" atomic is for both facts to live in one store that can commit them together. That is not a clever trick — it is the definition of a transaction, and it is the single insight underneath every real exactly-once implementation.
Auto-commit is the worst of both, on a timer
The default in most clients is to commit periodically in the background — every few seconds, from the fetch loop, with no knowledge of whether your handler finished. It is convenient, it is the default, and it gives you a guarantee that is neither of the two clean options.
Because the timer is unrelated to processing, it can fire *after* records are fetched but *before* they are processed, giving you at-most-once for those records — a silent gap. And because it fires only every few seconds, a crash also reprocesses everything since the last tick — duplicates. You get gaps and duplicates, and which one you get depends on where the timer landed relative to the crash. It is not a middle ground; it is a random choice between two failure modes.
It is also invisible. Auto-commit never errors, never logs, and produces a group whose lag looks perfect. Teams run it for years and attribute the occasional missing record to something else. The fix is a single configuration change plus an explicit commit call after processing, and it is one of the highest-value corrections available in a stream consumer.
| Strategy | On crash | Guarantee | Handler must be |
|---|---|---|---|
| Auto-commit on a timertypical | Gaps and duplicates, depending on timing | Neither cleanly | Idempotent, and you still lose records |
| Commit before processingprotocol | Records skipped silently | At-most-once | Nothing — and you have chosen loss |
| Commit after processing each recordprotocol | At most one duplicate | At-least-once | Idempotent |
| Commit after processing each batchprotocol | Up to one batch reprocessed | At-least-once | Idempotent |
| Offset in the same transaction as the outputassumption | Neither gap nor duplicate *in that store* | Effectively-once, scoped to the store | Nothing extra — the transaction does it |
| Broker transactions (read-process-write)typical | Atomic within the broker’s own topics | Effectively-once, scoped to the broker | Deterministic; no external side effects |
Where exactly-once actually comes from
Exactly-once is achievable, and it is achievable in exactly one way: make the offset part of the same atomic commit as the effect. Everything marketed as exactly-once is an instance of this, and the useful skill is identifying where the boundary of that atomicity sits.
If your consumer writes to a relational database, store the offset in a table in that database and update it in the same transaction as the business write. On restart, read the offset from your own table rather than from the broker. The transaction guarantees that either both happened or neither did, so there is no window. Within that database, processing is exactly-once — genuinely, not approximately.
If your consumer reads from a log and writes back to the same log, the broker’s own transaction support can commit the produced records and the consumed offsets atomically. That is exactly-once within the broker’s boundary: the consume-transform-produce loop is atomic, provided the transformation is deterministic and has no side effects outside it.
And if your consumer calls a third-party API, there is no shared transaction and there never will be. The best available construction is at-least-once plus an idempotency key that the third party honours. If they do not offer one, exactly-once is not on the menu and you are choosing between a duplicate and a loss on someone else’s behalf. Say which, in writing. See Exactly-Once Is a Scope, Not a Guarantee for the full treatment and Idempotent Is a Property of the Whole Effect, Not the Write for the mechanism.
1// The offset lives in OUR database, not in the broker. On restart we seek2// to what our own store says, because that is the store the effect is in.3async function onBatch(partition: number, records: Record[]) {4 await db.transaction(async (tx) => {5 for (const r of records) {6 await tx.upsert('order_view', project(r)) // the effect7 }8 await tx.upsert('stream_offsets', { // the position9 group: 'order-view',10 partition,11 offset: records.at(-1)!.offset + 1,12 })13 })14 // Committing to the broker too is optional and is only for lag visibility.15 // It is NOT the source of truth and must never be read on restart.16}17 18async function onAssign(partition: number) {19 const row = await db.get('stream_offsets', { group: 'order-view', partition })20 consumer.seek(partition, row?.offset ?? EARLIEST) // OUR offset wins21}Practical consequences: batching, rebalances, and the lag lie
Commit frequency is a throughput-versus-duplicate-window dial. Committing per record minimises duplicates and costs a round trip per record, which for a high-volume stream is unacceptable. Committing per batch of a few thousand is efficient and means a crash reprocesses up to that batch. Most systems land on per batch with a time bound, and then choose batch size by how many duplicates the downstream can absorb rather than by throughput alone.
A rebalance is a commit point whether you treat it as one or not. If you do not commit in the revocation callback, the new owner starts from your last periodic commit and reprocesses the difference — on every deploy, forever. Committing on revoke is a two-line change that removes most deploy-time duplicates. See Rebalancing: Everyone Stops So the Partitions Can Move.
Finally, be suspicious of lag. Lag is computed from the committed offset, not from what has been processed. A consumer that commits eagerly and processes slowly reports low lag while falling behind; a consumer that processes correctly but commits rarely reports high lag while being fine. Lag is a proxy for progress and it measures the commit, so if you want to know whether work is being done, measure completed records, not the gap between two numbers.
Key points
- Commit before processing risks silent loss; commit after risks duplicates. The offset store and the output store are different systems, so there is no third arrangement.
- Auto-commit on a timer gives you both failure modes depending on when the crash lands, and it is the default.
- Exactly-once means committing the offset in the same transaction as the effect — and it is scoped exactly to that transaction’s reach.
- Commit in the revocation callback, or every deploy reprocesses up to a full commit interval.
- Lag is computed from the committed offset, so it measures commits, not completed work.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The consumer fetches a batch of records starting at its current position.
- • It processes them, producing whatever effects the records imply.
- • It records a new position: to the broker’s offset store, or to its own store, or both.
- • On restart or reassignment, it resumes from the last recorded position for that partition.
- • Records processed but not recorded before a failure are processed again; records recorded but not processed are skipped.
- • The process crashes between processing and commit, producing duplicates.
- • The process crashes between commit and processing, producing a silent gap.
- • The commit call times out with unknown outcome — the position may or may not have advanced.
- • A rebalance revokes the partition before an uncommitted batch is recorded.
- • The consumer commits a position for a partition it no longer owns.
- • The offset store and the output store diverge because they were written separately and only one succeeded.
- • Silent gap under auto-commit: a downstream table is missing scattered records after each consumer crash. Group lag is zero, the consumer logs no errors, and the loss is discovered only by comparing record counts against the source.
- • Duplicate wave sized by the commit interval: every restart produces exactly one commit-interval worth of reprocessing. Idempotent handlers hide it; non-idempotent ones produce a small regular stream of double-counted rows.
- • Lag lying about progress: the operator sees lag near zero while a downstream view falls further behind. The consumer commits eagerly and processes into an unbounded internal buffer, so committed position has decoupled from completed work.
- • Offset ahead of output after a partial failure: the database write failed, the offset commit succeeded, and the records are unreachable — they are behind the committed position and will never be fetched again.
- • Offsets going backwards: two members briefly own the same partition across a rebalance and both commit, so the recorded position oscillates and a range of records is processed repeatedly.
- • The commit is coordination between the consumer and the offset store — one-way, and subject to the same ambiguity as any remote call.
- • Exactly-once requires the offset and the effect to agree, which means either one store (a local transaction, i.e. coordination *avoided*) or an atomic commit across two (Two-Phase Commit: Buying Atomicity With a Promise, which nobody wants here).
- • Broker-side transactions are a scoped form of this: the broker coordinates its own offset topic and its own output topics, and offers nothing outside that boundary.
- • With commit-after, no record is skipped; the cost is bounded reprocessing determined by the commit interval.
- • With commit-before, no record is processed twice; the cost is unbounded, silent, unrecoverable loss.
- • With offset-in-transaction, neither gap nor duplicate occurs *within the transactional store*, and nothing is guaranteed outside it.
- • Detect: compare source record counts to output record counts over a window. Lag will not show you a gap, and only this comparison will.
- • Contain: disable auto-commit before doing anything else; it is the mechanism actively creating the loss.
- • Recover: for a suspected gap, reset the group to a known-good offset and reprocess, relying on idempotence to absorb the overlap.
- • Reconcile: rebuild derived views from the source of truth for the affected range; the log can still supply the records if they are within retention.
- • Verify: counts match, and the commit strategy is explicit rather than defaulted. Then confirm duplicates are being absorbed rather than merely being invisible.
- • Completed-record throughput, measured in the handler — the only real progress metric, since lag measures commits.
- • Commit rate and commit failure rate, separately from processing errors.
- • Distance between processed offset and committed offset, which is the live duplicate window.
- • Source-to-output record-count delta per partition per window, which is the only detector for a silent gap.
- • Offset regressions per partition, which indicate two owners committing.
- • Always — every consumer has a commit strategy, and the default one is the worst available. Choosing deliberately costs nothing.
- • Offset-in-transaction is transformative wherever the consumer writes to a single transactional store, which is most view-building consumers.
- • Per-record commits on a high-throughput stream, where the round trip dominates the work and throughput collapses.
- • Building elaborate exactly-once machinery for a handler that is naturally idempotent, where a plain commit-after and an upsert are equivalent and far simpler.
- • Storing offsets in your own store without also updating the broker, which is correct but leaves standard lag monitoring blind — commit to both, and read from yours.
- • Idempotent handlers plus commit-after: the standard, simplest correct construction, and sufficient for the large majority of consumers.
- • Offsets stored in the output database inside the effect transaction, when the output is a single transactional store and you want a real guarantee.
- • Broker transactions for pure consume-transform-produce pipelines with no external side effects.
- • Accepting at-most-once explicitly for genuinely lossy streams (sampled telemetry), documented as a decision rather than inherited from a default.
Commit before or after — there is no third option
What people believe, and what is true
Auto-commit is a safe default.
It commits on a timer unrelated to your processing, so a crash can produce both a gap and duplicates. It is the only option that fails in both directions.
Exactly-once is impossible.
Exactly-once *delivery* is impossible. Exactly-once *processing* is achievable by committing the offset in the same transaction as the effect, and is scoped to that transaction.
Low lag means the consumer is keeping up.
Lag is measured from the committed offset. A consumer that commits eagerly and buffers internally reports low lag while falling further behind.
Committing more often is always better.
It shrinks the duplicate window and costs a round trip each time. On a high-volume stream, per-record commits can halve throughput and thereby increase lag.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Record your position after you have done the work, not before. Then a crash means doing some work twice, which idempotence handles — rather than skipping it, which nothing handles.
Practical
Disable auto-commit. Commit after processing each batch, and in the revocation callback. Size the batch by how many duplicates downstream can absorb. If the output is one transactional store, put the offset in that transaction and seek from it on assignment. Measure completed records, not lag.
Advanced
This is the same problem as the dual write in Atomicity Stops at the Process Boundary, seen from the consumer side: two independent stores, one atomic fact needed across both. The available answers are identical — collapse them into one store, or accept at-least-once and deduplicate at the boundary. Which is why "exactly-once" is always a claim about a boundary rather than about the world: it holds precisely as far as one atomic commit reaches, and the engineering skill is knowing where that edge is before you promise anything across it.
Apply it
- 🔧 Run a consumer with auto-commit, kill it repeatedly under load, and count the records that never reached the output. Then disable auto-commit and repeat.
- 🔧 Move the offset into the output transaction and prove that no crash timing produces either a gap or a duplicate in that database.
- ⚡ Group lag is zero and a downstream table is missing 0.3% of records. Diagnose from the commit strategy alone.
- ⚡ A consumer must call a third-party API that has no idempotency keys. Write down the guarantee you can actually offer and the failure the business must accept.
- 💬 When do you commit the offset, and what does each choice cost?
- 💬 Why is auto-commit worse than either explicit option?
- 💬 Build me exactly-once for a consumer that writes to Postgres. Now do it for one that calls Stripe.