Replay from the Log
Re-reading a retained event log versus recomputing from the raw layer — two recovery paths with different windows, different guarantees, and retention as the hard boundary on both.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A consumer has been broken for three days. Do you replay the log, rebuild from raw, or re-snapshot the source — and which of those is still available?
The dataset that has a gap in it, and everyone reading that dataset who currently believes the gap is a quiet period. The recovery has to restore what was missed without re-delivering what was not, which is a different requirement from "run it again".
Replay operates on offsets: a contiguous range of positions in one partition of one topic, per consumer group. Recomputation from raw operates on partitions of files. The two units do not align — one offset range can span several file partitions and vice versa — and confusing them is how a replay is asked to fix a range it cannot address (Offsets and Commits).
Reset the consumer group to an earlier offset and let it run. The log is durable and replayable, that is what it is for, and the pipeline downstream already handles the records. For a consumer whose outputs are idempotent and whose gap is inside retention, this is exactly right and it is the cleanest recovery available anywhere in this module.
The gap is older than the topic's retention. The records are not late, they are gone — deleted by a policy configured by someone who was thinking about disk usage rather than about recovery windows (Retention and Replay).
- The gap is older than the topic's retention. The records are not late, they are gone — deleted by a policy configured by someone who was thinking about disk usage rather than about recovery windows (Retention and Replay).
- The replay re-delivers records the consumer had already processed, and because the sink appends rather than merges, the recovery duplicates a period while repairing another (Upserts and Merges).
- Resetting the group offset repairs this consumer and moves the position for every consumer in the group, so a healthy partition owner starts re-reading history nobody asked it to (Consumer Groups and the Parallelism Ceiling).
- The replay runs at maximum speed against a downstream that was sized for the live rate, and the recovery becomes an outage of its own (Backpressure).
- The consumer is stateful, so replaying the input without resetting the state double-counts: the state already contains the effect of records that are about to be re-read (Stateful Stream Processing).
- Replay restores the events and not the enrichment: the transformation joins a dimension that has since changed, so the replayed records are reprocessed against today rather than against the moment they were first seen (Snapshot Tables).
What is actually happening
- A log replay re-reads records the broker still holds, in per-partition order, from a chosen position. What it restores is the input; whether that restores the output depends entirely on the consumer being deterministic and the sink being idempotent (Kafka as a Log, Not a Queue).
- Recomputation from the raw layer re-reads records you stored, in whatever layout you wrote them, for as long as you kept them. Retention is a storage decision under your control rather than a broker configuration, which is the main practical difference between the two paths (The Raw Landing Zone).
- The two have different windows and different guarantees. The log holds a short, ordered, complete window of every change including ones that never landed anywhere else; raw holds a long, unordered-across-files window of everything that successfully landed. A gap caused by an ingestion failure exists in the log and not in raw, which is exactly when replay is the only option (Ingestion Failure & Recovery).
- Retention is therefore a recovery-window decision, not a storage-cost decision, and it should be argued as one: the question is not how many days of disk you want to pay for, it is how long you are willing to take to notice a problem (Retention and Replay).
- Beyond both windows there is one path left: re-snapshot the source. That gives current state rather than history — every intermediate change between the gap and now is lost, and a fact table built from changes cannot be reconstructed from a snapshot of current state (Snapshot and Stream: the Bootstrap Problem).
- A replay is a reprocess, not a retry: the units being re-read were already processed once, so everything in Reprocessing vs Retrying applies, including that consumers will see numbers move (What Backfills Break).
Two paths back, and the wall behind both
When a range of data is missing or wrong, there are exactly three places it can come from again: the broker's retained log, your own raw layer, or the source system itself. They have different windows, different contents and different guarantees, and the choice between them is made for you by which windows are still open.
The important asymmetry is what each one contains. The log holds every change that was published, including ones that never reached your storage — which is precisely the case when an ingestion failure is the reason you are here. Raw holds everything that landed successfully, for much longer and much more cheaply. A source re-snapshot holds current state and no history at all.
Behind all three is retention, and it is a wall rather than a slope. Inside the window, recovery is a routine operation; one day outside it, the same recovery is impossible. Nothing about the failure changes at that boundary — only whether it can be fixed (Retention and Replay).
Replaying without causing a second incident
A replay run carelessly is a backfill run carelessly: it re-delivers into a live sink, at a rate the downstream was not sized for, moving offsets that other consumers depend on. The stages below are what make it boring, and each of them promises something specific.
The stage people skip is the second one. Resetting the live consumer group is one command and it is irreversible in the sense that matters — the group has now consumed those offsets again, other members are affected, and the outputs have already been written. A separate group reading into a staged target costs one configuration and makes the whole operation reviewable.
Note that the last three stages are the same as a backfill's, because from the moment records leave the log this *is* a backfill whose source happens to be a broker. There is nothing special about replay after the read (Planning a Backfill).
- 1Establish the range and its availability
Identifies the affected offset range per partition and checks the oldest available offset actually still covers it.
guarantees That the cheap path exists. If it does not, this answer arrives before any work is done rather than halfway through.
fails by Trusting the configured retention rather than the oldest available offset — a size-based limit routinely makes the real window shorter (Retention and Replay).
- 2Replay into a separate consumer group
Starts a second group at the chosen offsets, writing to a staged target no consumer reads.
guarantees The live group is untouched and every other member of it is unaffected.
fails by Being skipped in favour of resetting the live group, which repairs one consumer and forces history on all of them (Consumer Groups and the Parallelism Ceiling).
- 3Reset or rebuild consumer state
Brings any stateful operator to a checkpoint consistent with the replay start offset, or clears it.
guarantees That re-read records are folded into state once rather than twice.
fails by Replaying input against state that already contains its effect, which double-counts every aggregate (Checkpointing).
- 4Throttle
Caps the replay's consumption rate to something the downstream can absorb alongside live traffic.
guarantees That recovery does not become an outage of its own.
fails by Running at broker speed, which is far above live rate by design and is exactly what the downstream was never sized for (Backpressure).
- 5Validate the staged output
Reconciles the recovered range against the source, asserts uniqueness, and checks a period outside the range is unchanged.
guarantees That the differences are the gap being filled and nothing else (Validating a Backfill Before You Publish).
fails by Validating only that rows appeared, which is satisfied equally by a correct recovery and by a duplicated one.
- 6Publish and record
Merges or swaps the staged range into the serving table, then writes down what was replayed.
guarantees Atomicity per partition, and a record that these numbers changed and why.
fails by Appending, which duplicates any part of the range that was not actually missing (Upserts and Merges).
Stages one and two are the ones unique to replay. Everything from stage five onward is Planning a Backfill with a different input.
Which path, and what each one costs you
The decision is usually made by elimination rather than by preference — you take the first path whose window is still open. That is why the useful work happens long before the incident, when retention is being configured by someone who is thinking about storage.
The criterion that should drive retention is detection time: how long, typically, between a data problem starting and someone noticing. A platform that finds problems in three days and retains for two has decided, without discussing it, that its cheap recovery path will never be available for a real incident (Data Observability).
The last option is included because it is a real answer and teams are reluctant to say it out loud. Sometimes the data is gone, and the correct action is to annotate the period as incomplete rather than to fabricate a plausible reconstruction from current state (Data Incidents).
Which recovery windows are still open, and what does the range need to contain?
when The gap is inside broker retention, and it is a gap in *ingestion* — the records were published but never landed anywhere downstream.
cost Contention with live consumption, at-least-once redelivery that duplicates unless the sink merges, and a group offset that must not be reset in place. The cleanest option when it is available (Retention and Replay).
when The records landed successfully and the problem is downstream — a transformation bug, a bad model, a wrong join. Raw retention is usually far longer than the log's.
cost Compute proportional to the range, and nothing else. This is the calm path: no urgency, no contention with a live stream, fully stageable (Keeping Raw History: The Recovery Position and the Liability).
when Both windows have closed and the dataset represents current state — a dimension, a status table, anything where history is not the point.
cost Every intermediate change is lost. A dimension recovers fine; a fact table built from changes cannot be reconstructed this way at all (Snapshot and Stream: the Bootstrap Problem).
when A derived dataset, an extract or a warehouse table still holds the range and the upstream does not.
cost You are restoring from something further from the source, so any error already in that copy is now canonical. A genuine last resort and occasionally the only one (Source of Truth).
when No window is open and the data represents events that no longer exist anywhere.
cost A permanent hole in the series, and the discipline to mark it so that every future comparison spanning it is interpreted correctly rather than as a business trend (Dataset Documentation).
Broker retention semantics, the maximum retention available, and whether replay is expressed as an offset reset, a shard iterator or a subscription seek all differ by product and change over time. Treat the oldest actually available position as the source of truth for your recovery window rather than the configured retention, and verify current documentation before designing around a specific maximum.
How to build it
Most important first.
- Set retention from your detection time, not from disk. If a data quality problem is typically found in three days and retention is two, no replay will ever be available for the problems you actually have (Data Observability).
- Make every consumer's output idempotent before relying on replay as a recovery strategy. Replay guarantees redelivery, and redelivery into an appending sink is duplication (Upserts and Merges).
- Replay into a separate consumer group writing to a staged location, then validate and publish, rather than resetting the live group in place. It costs one more group and converts an irreversible operation into a reviewable one (Planning a Backfill).
- Reset stateful consumers from a checkpoint that is consistent with the offset you replay from, or reset the state entirely. Replaying input against stale state is the most common way a replay produces a worse number than the gap did (Checkpointing).
- Throttle the replay. A recovery that saturates the downstream turns one incident into two, and the throughput advantage of replaying at full speed is worth very little (Backpressure).
- Keep the raw layer as the long-horizon recovery position, with a retention argued against the questions you need to answer rather than against a default (Keeping Raw History: The Recovery Position and the Liability).
- Record what was replayed — topic, partitions, offset range, target, when — because a replayed range is a restatement and needs the same record as a backfill (Data Incidents).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A log replay guarantees the records the broker still holds are re-delivered in per-partition order. It guarantees nothing across partitions, and nothing about records outside retention, which are gone rather than delayed (Topics and Partitions).
- It guarantees at-least-once delivery on replay, exactly as on the original read. A replay that repairs a gap will also re-deliver records the consumer had already handled, unless the offset range is exact (At-Least-Once Delivery).
- Recomputation from raw guarantees only that what landed in raw is reprocessed. Anything that never landed — the ingestion gap itself — is not in raw and cannot be recovered from it, which is the case that decides between the two paths (Missing Rows).
- Neither guarantees that reprocessing reproduces the original output. That requires the transformation to be deterministic and its reference data to be pinned, which is a property of your code rather than of the log (Idempotent Data Pipelines).
- A re-snapshot guarantees current state and explicitly loses history: the sequence of changes between the gap and now is not recoverable from it at all.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Reconcile the recovered range against the source for a closed period after the replay completes. It is the only check that distinguishes "the gap is filled" from "the consumer ran again" (Reconciliation).
- Assert uniqueness on the business key across the replayed range, because a replay whose offset boundaries were approximate will have re-delivered records that were already processed (Deduplication).
- Both miss the records that were never in the log to begin with — a producer that failed before publishing leaves no trace anywhere downstream, and no amount of replay recovers what was never sent (CDC Failure Modes and the Retention Deadline).
- A replay competes with live consumption for the same downstream capacity, so recovery and freshness are in direct tension for the duration — which is the argument for a separate group and a throttle rather than resetting in place (Consumer Groups and the Parallelism Ceiling).
- The freshest recovery is the one that starts soonest: every hour of detection delay consumes an hour of the retention window, and past it the cheap path disappears (Freshness Monitoring).
- Recomputation from raw has no freshness urgency at all, which is exactly what makes it the calm path: it can be scheduled, staged and validated, because the data is not going anywhere (Full Refresh vs Incremental).
- Replaying old records through current code runs today's logic over yesterday's payloads. If the schema evolved, the consumer must still be able to read the old shape, which is what backward compatibility in the registry is for (Backward Compatibility).
- This is the strongest practical argument for a schema registry with enforced compatibility: retention defines how far back you may need to read, and any incompatible change made inside that window makes part of the log unreadable (Schema Registry).
- A topic that was repartitioned within the retention window has records for one key in two partitions, so a replay restores them without their original relative order (CDC Ordering and Transaction Boundaries).
- This lesson is the recovery mechanism, so its own recovery question is what to do when it is unavailable: raw first, then a source re-snapshot, then an acknowledgement that the history is gone and the period must be annotated rather than fixed (Data Incidents).
- A replay that went wrong is recovered like any other bad publish: restore the target range from a snapshot and replay again into staging, with exact offsets this time (Rolling Back Data).
- The strategic recovery position is the pair — a log long enough to cover detection time, and a raw layer long enough to cover everything else. Platforms that keep only one of the two discover which one they needed during the incident (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- The gap falls outside retention, so the cheap path does not exist and nobody knew until they looked.
- Replay duplicates a period because the sink appends and the offset range was approximate (Duplicate Rows).
- Resetting a shared consumer group's offsets, repairing one consumer and forcing history on every other member.
- A replay saturating the downstream and causing a second incident during the recovery from the first (Backpressure).
- Stateful consumers replayed without resetting state, double-counting the records they had already folded in (Streaming State).
- Retention extended after the incident to make replay possible next time, which protects against the last failure and not the next one — the mitigation aimed backwards (Retention and Replay).
- "The log is durable, so we can always replay." You can always replay what is still there. Retention is a deletion policy, and past it the records are gone from the log as thoroughly as if they had never been written (Retention and Replay).
- "Replaying is safe because the log is immutable." The log is immutable; your sink is not. Replay re-delivers, and re-delivery into an appending sink duplicates (Upserts and Merges).
- "We keep raw data, so we do not need log retention." Raw contains what successfully landed. The failures worth recovering from are precisely the ones where nothing landed, and those exist only in the log (Ingestion Failure & Recovery).
- "We can re-snapshot the source if we lose events." A snapshot gives current state. A history of changes — what a fact table is built from — cannot be reconstructed from it, and the intermediate states are gone (Event vs Snapshot Modeling).
- Retention is simultaneously a recovery argument and a privacy obligation, and they pull in opposite directions: the log is a copy of personal data held for the length of the window, and a deletion request must reach it or be documented as unreachable (Data Retention).
- A replay re-delivers records that may include entities deleted since, so any replay of personal data has to re-apply the suppression list rather than faithfully restoring what the log holds (Deletion Requests).
Operating it
- Consumer lag per group and partition, which is what turns "we have a gap" into "we have a gap of this size, here" (Consumer Groups and the Parallelism Ceiling).
- The oldest available offset per partition against the current time — the actual, measured recovery window, which routinely differs from the configured retention because of size-based limits (Pipeline Metrics).
- Raw layer coverage per source and day, so the second recovery path's window is known before it is needed rather than during an incident (Data Observability).
- Detection time for data incidents as a tracked metric, because it is the number that should be setting retention and almost never is (Data Incidents).
- At 10x throughput a size-based retention limit bites long before the time-based one, so the real recovery window shrinks silently as volume grows. Measuring the oldest available offset is the only way to know (Retention and Replay).
- At 100x, replaying a long range at full speed is not survivable for the downstream and recovery becomes a throttled, staged, scheduled operation — which is to say it becomes a backfill with a log as its source (Planning a Backfill).
- More consumers make offset resets more dangerous, because the blast radius of a group-level reset grows with the number of things reading that group (Consumer Groups and the Parallelism Ceiling).
- Log retention costs bytes held on the broker's storage for the whole window, for every topic, whether or not anything ever replays. It is the most visible line and the one most often cut (What Actually Drives Data Platform Cost).
- Raw retention costs bytes on object storage, which is the cheapest tier in the platform and is why raw is the long-horizon path while the log is the short one (Storage Lifecycle).
- A replay costs the downstream compute of reprocessing the range, plus whatever contention it imposes on live consumption — the same shape of collateral cost as a backfill (What Backfills Break).
- The cost of retention being too short is a recovery that is impossible, paid as permanent missing data. That expected cost belongs in the retention argument and almost never appears in it.
- Long retention buys a long recovery window and costs broker storage continuously. It is insurance, and the premium is paid whether or not there is ever a claim.
- Replaying into a separate group and a staged target is slower and safer than resetting in place, and it costs an extra consumer plus a validation step — the same trade as staging a backfill and correct for the same reason.
- The raw layer duplicates what the log already holds for the length of the shorter window. That redundancy is the point: the two fail differently, and an ingestion gap exists in exactly one of them.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka replays by resetting a consumer group's committed offsets and retains by time or size per topic, with compacted topics keeping only the latest record per key; Kinesis retains per stream up to a fixed maximum and replays by shard iterator; Pub/Sub replays by seeking a subscription to a timestamp or snapshot. The recovery unit and the retention semantics differ in every one.
- GENERALThe structural point — two recovery paths with different windows, and retention as the hard boundary on both — holds for any architecture with a durable input log and a stored raw layer, regardless of which products implement them.
- SIMULATEDThe behaviour of the
replay-from-logmitigation insrc/de/sim/pipeline.ts— it repairs a CDC gap when the records are still retained and cannot help past the window — is a deterministic teaching model pinned byscripts/de-sim.test.ts, not a measurement of any broker.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the log as a primitive — why an ordered, replayable, durable sequence is the foundation replication and state machine recovery are built on, and what per-partition ordering does and does not give you.
- — DevOps / Production Engineering owns recovery objectives as an operational commitment: how much data you can afford to lose and how long you can afford to take, which is the same argument this lesson makes about retention.