Retention and Replay
Retention is not a storage setting. It is the maximum age of a bug you can fix by replaying instead of reconstructing — a recovery-window decision that happens to be paid for in disk.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A transformation bug has been producing wrong numbers for three weeks. Can we fix it by reprocessing, or do we have to reconstruct?
Everyone who will one day need history they did not know they needed: the analyst whose metric was wrong since a deploy, the team building a new derived dataset from scratch, the auditor asking for a period to be recomputed. What they need from the log is that the records for the period in question still exist, and that is decided entirely by a number somebody set once.
The unit of retention is a segment, not a record — records age out in whole files, so the boundary is approximate. The unit of replay is a (group, partition, offset) position, which is what makes "reprocess from here" a precise instruction (Kafka as a Log, Not a Queue).
Set retention to whatever keeps disk usage comfortable — a few days is the usual default — and treat it as an operational hygiene setting owned by whoever runs the cluster. Nothing is obviously wrong for months, because nothing needs to be replayed for months.
A bug is found three weeks after it shipped. Retention is four days. The records that would fix it aged out eighteen days ago, and the only remaining path is to reconstruct history from whatever downstream tables still hold, which is a project rather than a re-run (Reprocessing vs Retrying).
- A bug is found three weeks after it shipped. Retention is four days. The records that would fix it aged out eighteen days ago, and the only remaining path is to reconstruct history from whatever downstream tables still hold, which is a project rather than a re-run (Reprocessing vs Retrying).
- A consumer is paused for a long weekend of maintenance. It restarts, its committed offset no longer exists, the reset policy picks a valid position, and the gap is skipped silently. The restart looks completely normal (Consumer Groups and the Parallelism Ceiling).
- A cost review halves retention. Nothing breaks and nobody notices, because the capability that was cut is only exercised during incidents, and the next incident is months away.
- Someone enables compaction on a topic to keep it small, and a downstream that reconstructs a state machine from the full sequence of changes now sees only the latest record per key. Its output is not wrong-looking — it is subtly missing every intermediate transition (What a CDC Event Contains).
- A replay is finally run and produces different numbers than the original, because the transformation reads a dimension table that has been updated in place since. The replay is not a re-run; it is a new computation against a different world (Idempotent Data Pipelines).
What is actually happening
- Retention is a per-topic policy — by time, by partition size, or both — evaluated against whole segments. When a segment is entirely older than the window it is deleted; records inside it are never removed individually, which is why "delete this record" is not an operation the structure supports.
- Consumption has no effect on retention whatsoever. This is the property that makes replay possible, and it is the exact inverse of a queue, where consumption is the only thing that removes anything (Message Brokers: Log-Shaped and Queue-Shaped).
- A replay is a write to the consumer group's committed offsets, nothing more. Point the group at an earlier position — by offset, or by timestamp using the segment time index — and it reads forward from there. The broker does no work beyond serving bytes it already had (Replay from the Log).
- Compaction is a different policy, not a cheaper retention. It retains the most recent record per key and discards earlier ones, producing a snapshot of current state. It is the right choice for a changelog backing a state store and the wrong one for anything that needs the sequence (Stateful Stream Processing).
- Replay determinism is a property of your pipeline, not of the log. If the transformation depends on
now(), on a mutable lookup table, or on a non-idempotent merge, replaying the same records produces a different result — and the log will faithfully hand you the same records either way. - The recovery window a platform actually has is the maximum of the broker's retention and the raw landing zone's retention. The second is usually far longer and far cheaper, which is why a raw landing writer is the standard mitigation for a short broker retention (The Raw Landing Zone).
The window, and what falls off the back of it
Draw the log as a line with two moving boundaries. New records arrive at the right; the retention boundary advances from the left, deleting whole segments as they age past the window. Every consumer sits somewhere between the two, and the distance from a consumer to the left-hand boundary is how much slack it has before its position becomes invalid.
What makes this worth drawing is that both boundaries move. A consumer that is not falling behind at all can still lose its position, because the left boundary is advancing at the append rate regardless. A consumer that stopped on Friday is not "a bit behind" on Tuesday — it may be past the boundary, and the restart that follows will look entirely routine.
The decision below is the one this lesson exists to reframe. Retention is normally chosen as a disk number. Choose it instead as an answer to "how old can a mistake be and still be fixable by replay", and the options separate cleanly by what kind of error each protects against.
How old is the oldest mistake you must be able to fix by replaying rather than reconstructing?
when The topic drives operational behaviour only, has no analytical consumer, and any error is detected by an alert within minutes.
cost No analytical recovery at all. Acceptable only if nothing downstream needs history and a raw landing zone exists for anything that might.
when Quality checks run daily and someone reviews the outputs daily, so detection lag is measured in days.
cost A bug found at a monthly review is unrecoverable from the log. This is the most common setting and the most common source of the phrase "we cannot reprocess that".
when Detection lag genuinely runs to weeks — metrics reviewed at period end, seasonal comparisons, a consumer that is deployed intermittently.
cost Retained bytes multiplied by the window and by the replication factor, paid continuously, plus a matching obligation to keep old schemas readable.
when Almost always. Days in the broker for operational replay, years in object storage for everything else.
cost An extra consumer to operate and monitor, and a second recovery path that is a reprocess from files rather than an offset reset — different tooling, and it must be exercised or it will not work when needed (The Raw Landing Zone).
when The topic is a changelog whose only purpose is to rebuild current state per key, and no consumer needs the sequence.
cost History is destroyed by design. Any future consumer that needs intermediate transitions cannot be built from this topic, and nothing will warn them (Stateful Stream Processing).
partition 0, viewed as a line
[ deleted, gone ][ RETAINED WINDOW .............................. ]
^ ^ ^
oldest readable consumers log end
offset 88 200 | | offset 91 274
| |
warehouse-loader @ 91 200 ---+ |
reindex-2026 @ 88 640 -------+
slack: 440 records
Both boundaries move:
- log end advances at the append rate
- oldest readable advances as segments age out
A consumer does not have to fall behind to lose its position.
The floor rises underneath it.
When oldest-readable passes a committed offset:
reset to earliest -> silently re-reads a large range
reset to latest -> silently skips everything in between
both look like a normal restart in the logsCompaction keeps state, not history
Compaction is offered as a way to keep a topic small, and it is, which is why it gets enabled for the wrong reason. What it actually does is change what the topic *is*: from a record of everything that happened to a snapshot of where things ended up.
For a changelog backing a state store, that is exactly right and genuinely elegant — restart the processor, read the compacted topic from the beginning, and you have current state per key with no separate snapshot mechanism. For anything reconstructing a sequence, it is silent destruction: the intermediate transitions are gone, no consumer can tell which ones were removed, and the records that remain are individually valid.
The distinction to hold onto is that these are two different products of the same storage. If you need both, publish both — a compacted state topic and a time-retained history topic — and let the cost of the second be an explicit decision rather than a discovery.
Enable compaction on the orders topic because it is growing. Each key keeps its most recent record; older records for that key are rewritten away by a background process. The topic gets smaller and every current-state consumer keeps working perfectly.
Keep time-based retention sized to your detection lag, and have the topic's first consumer write every arriving record untouched to object storage. The topic stays bounded, the history is complete, and the long window is paid for at object-storage rates rather than at replicated-broker-disk rates.
Compaction and time retention answer different questions. Compaction answers "what is the current value for each key" and destroys the sequence to do it; time retention answers "what happened during this period" and pays storage to keep it. A consumer reconstructing a state machine, computing durations between transitions, or auditing a change needs the sequence, and compaction removes it without any signal that it did. Sizing the broker window from detection lag and offloading the rest to cheap storage buys the full history for a fraction of the cost, which is why the size problem compaction appeared to solve was the wrong problem.
What a replay actually re-derives
A replay is often described as "re-running the pipeline", which understates it in a way that causes incidents. What is being re-run is today's code, against yesterday's records, using today's lookup tables and today's clock. Three of those four are not the same as they were during the original run.
The chain below is the sequence a replay actually walks, with what each node can corrupt during the replay specifically. Read it before running one. The recurring theme is that every node which is not a pure function of its inputs is a place where the replay diverges from the original — and nothing in the pipeline compares the two, so divergence is silent.
The discipline that makes replay trustworthy is small and non-negotiable: bound the range explicitly, run in a separate consumer group, write to a scratch destination, reconcile against the source for the replayed period, then publish atomically. Every step of that exists because of a specific way replays have gone wrong (Planning a Backfill).
- Committed offsets for the replay group
holds The position the reprocess starts from, per partition.
could corrupt Resetting the *production* group instead of a scratch one, so the live pipeline reprocesses history and simultaneously stops keeping up with the present.
↑ reads from - Retained records in the partitions
holds The original records, unchanged, for as long as retention allows.
could corrupt A range that is partly expired. The replay reads what survives, succeeds, and produces a short result that looks complete (Missing Rows).
↑ reads from - Deserialisation and schema resolution
holds The mapping from stored bytes to the shape today's code expects.
could corrupt Old records written under a previous schema. A strict reader fails loudly, which is fine; a permissive one coerces to null, which is not (Nullability & Defaults).
↑ reads from - The transformation
holds The business logic, at its current version.
could corrupt Any reference to
now(), to a mutable dimension table, or to a random or non-deterministic function. Each makes the replay a different computation than the original (Idempotent Data Pipelines).↑ reads from - Dimension and lookup joins
holds Today's values for attributes that have changed since.
could corrupt Assigning current attributes to historical facts — a customer's current tier applied to an order placed before they changed tier (Slowly Changing Dimensions).
↑ reads from - Scratch destination
holds The rebuilt output, not yet visible to consumers.
could corrupt Writing to the live table instead, so consumers read a partially rebuilt dataset while the pipeline reports success (Atomic Publish).
↑ reads from - Validation and reconciliation
holds The evidence that the rebuild is right before anyone sees it.
could corrupt Being skipped, which is the normal outcome under incident pressure and the reason a bad backfill is usually discovered by a consumer (Validating a Backfill Before You Publish).
↑ reads from - Atomic publish
holds The swap that makes the rebuilt period visible in one step.
could corrupt A publish that overlaps periods it was not supposed to touch, overwriting data that was already correct (What Backfills Break).
Four of these eight nodes can produce a wrong result while every job succeeds. The log's contribution to the replay is the second node only — the rest is your pipeline's determinism, and no broker setting improves it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A bug is found and the affected range is older than retention. | The replay runs, succeeds, and covers only part of the period. Row counts are short and nothing flags it. | Records aged out at segment granularity. The consumer reads from the oldest surviving offset without any signal that earlier records existed. | Reprocess from the raw landing zone instead of the topic, and reconcile the full range against the source before publishing. Then lengthen the recovery window — in cheap storage, not necessarily in the broker (Reconciliation). |
| A paused consumer restarts after its committed offset expired. | It reports healthy and reaches near-zero lag within minutes. | The offset reset policy chose earliest or latest. Either the gap was skipped or a large range was re-read, and both are normal-looking startups. | Alert on offset-reset events as data incidents; alert on oldest-retained-age versus maximum lag so the situation is caught while it is still preventable (Data Incidents). |
| A replay is run and the recomputed numbers differ from the original. | Two versions of a closed period, both produced by successful pipeline runs, with no way to say which is right. | The transformation is not a pure function of the records — a now(), a mutated dimension, a non-idempotent merge. | Make the transformation deterministic, and run a scheduled replay-equivalence test so this is discovered on an ordinary Tuesday rather than during an incident (Reprocessing vs Retrying). |
| Compaction is enabled to reduce topic size. | A downstream that computes time-between-states starts producing implausible durations, gradually, as compaction catches up. | Intermediate records per key are being discarded. The remaining records are valid and the sequence is not. | Split the topic: a compacted state topic and a time-retained history topic. Treat "enable compaction" as a contract change requiring consumer sign-off (Contract Enforcement). |
How to build it
Most important first.
- State the recovery window as a commitment — "any bug discovered within N days can be fixed by replay" — and derive retention from it rather than from a disk budget. Set N from your worst realistic detection lag: if checks run daily but a metric is reviewed monthly, a four-day window means the monthly review can never be fixed by replay (Data Quality).
- Land raw records to object storage as the first consumer of every topic. It converts a broker retention window measured in days into a history measured in years, at a fraction of the cost per byte (The Raw Landing Zone, Object Storage).
- Alert on the gap between the oldest retained record and the furthest-behind consumer, so impending loss is visible while it is still preventable. This is the single most valuable alert in the module and almost nobody has it.
- Make every transformation a pure function of its inputs — no
now(), no in-place-mutated lookups, no non-idempotent merges — because a replay you cannot trust is not a recovery mechanism (Idempotent Data Pipelines). - Replay into a scratch destination, validate against the original, then publish atomically. Replaying over a live table means consumers read a half-rebuilt state and the pipeline reports success throughout (Validating a Backfill Before You Publish, Atomic Publish).
- Never enable compaction on a topic that feeds anything reconstructing a sequence. If both are needed, publish twice: a compacted state topic and a retained history topic.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Records are available for at least the configured window and are deleted at segment granularity, so the true boundary is a little older than the configured one and is never a little newer.
- Consumption never removes a record. Fan-out and replay are consequences of this and not of any additional feature.
- A replay of a retained range returns the same records in the same per-partition order as the original read. It does not return the same interleaving across partitions, so an order-sensitive downstream can produce a different result (Topics and Partitions).
- On a compacted topic, only the latest record per key is guaranteed to survive. Intermediate records may be gone, and there is no way for a consumer to detect which were removed.
- Nothing guarantees that a replay reproduces the original output. That is a property of the transformation, and the log has no view into it (Reprocessing vs Retrying).
- Past retention there is no guarantee and no recovery. This is the one failure in the module with no mitigation after the fact.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Run a replay equivalence test on a schedule: reprocess a closed historical window into a scratch location and compare row counts and summed measures with what is in production. If they differ, the pipeline is not deterministic and your recovery plan does not work.
- It misses non-determinism that happens to cancel out across the aggregate you compare, and it says nothing about periods outside the window you tested — including the oldest retained data, which is exactly where a replay is most likely to be needed and most likely to hit a schema it cannot read.
- Also verify the recovery window itself: assert that the oldest retained record is at least as old as the stated commitment. A retention setting that was quietly halved is otherwise invisible until an incident (The Freshness SLO).
- Retention has no effect on freshness — it decides how far back you can go, not how quickly a record arrives. Conflating the two is common because both are discussed in units of time.
- A replay competes for consumer capacity with live traffic if it runs in the same group, so the cheap pattern is a separate group writing to a scratch destination, which leaves live freshness untouched (Consumer Groups and the Parallelism Ceiling).
- A long replay temporarily degrades the freshness of whatever the replaying consumer feeds, because it is reading history rather than the head of the log. Plan the destination accordingly rather than being surprised by it.
- Retention is the schema-compatibility horizon. Consumer code must be able to read the oldest retained record, because a replay will hand it one, so lengthening retention lengthens the period over which old schemas must remain readable (Backward Compatibility).
- Shortening retention silently shortens that obligation, which is the one respect in which a shorter window is easier. It is not a reason to shorten it.
- A replay across a schema change reads both shapes in one pass. Any transformation that was written after the change and never tested against the older shape will fail — or worse, coerce — on the oldest records (Schema Evolution).
- On a compacted topic a key's last record persists indefinitely, so records written under a schema years old may still be present. Compaction converts a retention window into an unbounded compatibility obligation.
- Within retention: reset a scratch consumer group to the start of the affected range, reprocess into a scratch destination, validate, publish atomically, retire the scratch. This is the whole procedure and it should be routine rather than heroic (Planning a Backfill).
- Bound the replay to the affected range rather than replaying everything. A full replay costs the whole history and risks overwriting periods that were correct (What Backfills Break).
- Past retention: the raw landing zone is the recovery path, and if there is none, the only remaining options are reconstruction from downstream tables or an admission that the period cannot be recomputed.
- After a replay, reconcile the replayed range against the source before publishing. A replay that silently dropped records because of a schema failure produces a confident, complete-looking, short result (Reconciliation).
What can go wrong
- Retention shorter than the detection lag for the errors it is meant to protect against — the default state of most clusters, and invisible until the day it matters.
- A consumer's committed offset ageing out, followed by an automatic reset that silently skips or re-reads a large range.
- Compaction enabled on a topic whose consumers need the sequence, removing intermediate records with no signal to anyone.
- A replay that is not deterministic, producing a different answer than the original run, with no mechanism anywhere that notices the two disagree.
- A replay run in the production consumer group and written straight over the live serving table, so the pipeline stops keeping up with the present while consumers read a partially rebuilt dataset that never existed as a consistent whole (Atomic Publish).
- Retention lengthened to improve recovery without anyone checking that current consumer code can still read the oldest records, so the recovery window is nominal rather than real.
- "Retention is a storage setting." It is the maximum age of a bug you can fix by replaying. Frame the conversation that way and the number changes (Kafka as a Log, Not a Queue).
- "We have replay, so we can recover from anything." Replay recovers what the log still holds, and only if the transformation is deterministic. Both conditions fail more often than teams assume.
- "Compaction gives us history in less space." Compaction keeps the latest record per key and throws the rest away. It is the opposite of history.
- "A replay is just re-running the pipeline." A replay re-reads old records against today's code, today's lookup tables and today's clock. Unless all three are held constant it is a new computation (What Backfills Break).
- "Longer retention is always safer." Longer retention is a longer compatibility obligation and a larger surface for personal data you are required to delete. It is a trade, not a strict improvement (Data Retention).
- "We can extend retention when we need it." Retention applies going forward. Extending it tomorrow does not bring back what expired yesterday.
- Retention on a topic carrying personal data is a data-retention decision subject to the same rules as any other store, and it is usually set by an infrastructure team that has never been shown the classification (Data Retention).
- A deletion request cannot be honoured by editing an immutable log. The workable designs are a retention window short enough that deletion happens by expiry, personal data held behind a token resolved elsewhere, or crypto-shredding the key that decrypts the payload (Deletion Requests).
- Compacted topics keyed by a person retain their most recent record indefinitely, so an optimisation for a state store becomes an unbounded retention nobody signed off (PII in Pipelines).
- Every replay re-delivers historical personal data to every consumer in the replaying group, including any that was supposed to have been deleted downstream. Replay is a re-distribution event and deserves to be treated as one.
Operating it
- Oldest retained record age per partition, asserted against the stated recovery-window commitment. This turns a config value into a monitored promise.
- That same number plotted against maximum consumer lag. Convergence is the leading indicator of unrecoverable loss and has no other detector (The Backlog Arithmetic: Four Levers and a Drain Time).
- Offset-reset events per group, treated as data incidents. A reset means a consumer's position was invalid, which means records were skipped or re-read (Data Incidents).
- Replay runs as first-class pipeline events, with the range replayed, the destination and the validation result, so history is auditable rather than remembered (Data Lineage).
- Retained bytes per topic against its retention setting, which is how a quietly-shortened window becomes visible before an incident finds it.
- At 10x append rate the same retention window costs 10x more, which is when retention comes up in a cost review and when the recovery window gets cut without being framed as such.
- At 100x, keeping a long window in the broker becomes genuinely unreasonable and the raw landing zone stops being a nice-to-have. The correct response is to move the long window to cheap storage, not to shorten it (Object Storage).
- Replay cost scales with the window, so at high volume a "just replay everything" plan stops being feasible and bounded, range-limited backfills become mandatory (Planning a Backfill).
- Consumer count does not affect retention cost, which is the one thing that stays flat here.
- Retained bytes = append rate × retention window × replication factor, paid continuously. Doubling the recovery window doubles this line for every byte, including the overwhelming majority nobody will ever re-read.
- Raw landing storage is the same history at a much lower cost per byte, with no replication factor multiplier and lifecycle tiering available. Almost always the cheaper way to buy a long recovery window (Storage Lifecycle).
- A replay costs read bandwidth plus the full downstream compute of the window replayed. Its cost is set by the size of the window, never by the size of the bug (Compute Waste).
- Compaction costs background rewrite work in exchange for a smaller topic, and it buys that with the history it discards — a cost that appears in no cost report at all.
- A longer retention window buys a longer recovery window and costs storage continuously, plus a longer period over which old schemas must remain readable.
- Moving the long window to a raw landing zone buys most of the same capability far more cheaply and costs an extra hop, an extra consumer to operate, and a replay path that is a reprocess from files rather than an offset reset.
- Compaction buys a small topic and a usable state snapshot and costs the history — a good trade for a changelog backing a state store, a catastrophic one for anything that reconstructs a sequence.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka retains by time or partition size per topic and offers compaction as an alternative policy, with consumer-seekable offsets. Kinesis has a bounded retention window with no compaction and positions addressed by shard iterator. Pub/Sub retains per subscription with a seek-to-timestamp rather than an offset. "How far back can we go" therefore has a different answer and a different mechanism in each.
- GENERALThe framing — retention is a recovery window, and its length is the maximum age of a fixable mistake — applies to any immutable-history store, including a raw landing zone, a versioned table format's snapshot expiry, and database backups. The number is different in each; the argument for choosing it is identical.
- ORG-SPECIFICWhether four days or forty is right depends on how long errors take to be detected in your organisation, which is a function of review cadence and quality coverage rather than of the technology. A team with daily reconciliation needs a far shorter window than one whose metrics are checked at month end.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what a replicated log actually promises about the records you are relying on still being there — in-sync replicas, acknowledgement levels, and what an unclean leader election can silently remove from the tail.
- — DevOps / Production Engineering owns the change-management side: retention is configuration, configuration drifts, and a recovery window that is not asserted in monitoring is a recovery window that will be shortened by someone optimising a disk alert.