IngestionGENERALBROKER-SPECIFICTOOL-SPECIFIC

Ingestion Failure & Recovery

What to do when an extract fails, a connector stalls or a consumer falls behind — and how to tell, quickly, whether the data is late or gone.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

Ingestion has been broken for some hours. Which of the missing data is recoverable, from what, and what will re-running it break that currently works?

Who needs this

The engineer who was paged and the analyst who will read the table afterwards. The first needs to know whether to hurry — is this a latency problem or a countdown against a retention window. The second needs to know, later, whether the affected period was repaired or merely resumed, because "we fixed the pipeline" and "we recovered the data" are different sentences and only one of them is about the data.

What one row is

The unit of recovery is whatever the ingestion design made addressable: a window for batch, an offset range per partition for streaming, a key range for a snapshot repair. If none of those is addressable — because the window came from the clock or the offsets were not tracked per partition — then the unit of recovery is "everything", and that is the actual failure being discovered during the incident.

The obvious build

Fix the cause, restart the pipeline, watch it go green, close the incident. The pipeline is healthy again and the dashboards update, so the problem is over.

Why it breaks

The pipeline resumed from the present. The hours it was down were never re-requested and the gap is now permanent, sitting invisibly in the middle of a table that is otherwise complete (Incremental Extraction).

How it breaks with real data

The pipeline resumed from the present. The hours it was down were never re-requested and the gap is now permanent, sitting invisibly in the middle of a table that is otherwise complete (Incremental Extraction).
The pipeline caught up by re-reading a wide range and appending it, so the affected period is duplicated rather than missing — and the total is high rather than low, which nobody questions (Duplicate Rows). Worse, it ran every missed window at once, and the load took down the source that had just recovered (Retry Storms: The Load You Generated Yourself).
The streaming consumer resumed and started from the oldest available offset, because its committed position had aged out of retention. Everything between its old position and that point is gone and the resumption looked completely normal (Retention and Replay).
The backfill of the missing hours ran against the source *as it is now*, silently overwriting the affected period with current values for rows that have since changed (What Backfills Break).
The recovery re-ran ingestion but not the transformations built on it, so raw is repaired and every serving table still reflects the gap (Reprocessing vs Retrying).
Nobody recorded that the period was affected, so six months later an analyst investigating a dip has no way to know that the dip was an incident rather than a business event (Data Incidents).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Every ingestion failure is one of three kinds and the response differs completely: late (the data still exists everywhere and simply has not moved), gapped (the data exists at the source but the pipeline will never ask for it again), or lost (the data no longer exists anywhere you can read). Naming which one you have is the first and most valuable minute of the incident.
The transition from late to lost happens at a retention boundary — the broker's retention, the source's log retention, the API's change-history window. Before it, recovery is a replay; after it, recovery is not available at any price. That boundary is the clock the incident is actually running against (CDC Failure Modes and the Retention Deadline).
The transition from late to gapped happens when a bookmark advances past unlanded data, which is precisely the failure mode of committing a position before the write is durable, and of clock-derived windows that move on regardless of what succeeded (Offsets and Commits).
Recovery is a re-run, and a re-run is safe exactly to the degree the write is idempotent. Everything about incident recovery reduces to whether that property was built in advance, because it cannot be added during the incident (Idempotent Data Pipelines).
Catch-up is itself a load event. A pipeline that was down for six hours has six hours of work to do at once, against systems that are frequently still fragile. Uncontrolled catch-up is one of the most reliable ways to convert a recovered outage into a longer one (Without Jitter, Every Client That Failed Together Retries Together).
Recovering ingestion does not recover the platform. Raw is upstream of transformations, models and marts, and every one of them needs to be reprocessed for the affected range before any consumer sees a correct number (Data Lineage).

Late, gapped, or lost

The first minute of an ingestion incident should be spent on classification, because the three cases have almost nothing in common. Late means the data still exists at the source and in every intermediate system, and the only problem is that it has not moved — this needs patience and capped catch-up. Gapped means the data exists at the source but the pipeline's own bookkeeping has moved past it, so it will never be requested again unless a human requests it. Lost means it no longer exists anywhere readable.

Getting this wrong wastes the only resource the incident has. Teams routinely treat a gapped incident as a late one — restart the pipeline, watch it turn green, close the ticket — and the gap becomes permanent. Teams equally often treat a late incident as a lost one and begin an expensive backfill from a source that would have caught up on its own.

The clock that matters is the retention boundary, because it is where late becomes lost. If a consumer is twenty hours behind on a topic with a two-day retention, there are roughly twenty-eight hours before a recoverable problem becomes an unrecoverable one, and every decision in the incident should be made with that number visible.

Classify first, then act

Trigger	Symptom	Cause	Response
Extract fails on a transient source error	One run red, next run green, a window quietly absent.	Late becoming gapped: the schedule computed the next window from the clock rather than from the last completed one.	Late while the source still has the rows. Re-request the specific window by parameter; then change the scheduling so the next outage queues the window instead of skipping it (Batch Ingestion).
Consumer lag growing steadily	Dashboards getting older. No errors anywhere.	Consumer throughput below production rate — under-provisioned, rebalancing, or blocked on a slow write.	Late. Add consumers up to the partition count and watch lag against retention. If lag is still growing at the ceiling, partition count is the constraint (Topics and Partitions).
Consumer restarts and lag drops to zero instantly	Everything looks healthy. Volume for the outage period is a fraction of normal.	Lost: the committed offset had expired, and the reset policy jumped to the latest available position.	Nothing on the streaming side can recover it. Check whether the source system can still be extracted for the period, and record the gap if not (Missing Rows).
Connector down longer than the source's log retention	Connector resumes and requests a snapshot, or fails to start at all.	Lost: the change log segments covering the outage were reclaimed by the source.	A fresh snapshot restores current state and cannot restore the changes in between. Stitch it explicitly and record that the period holds state, not transitions (Snapshot and Stream: the Bootstrap Problem).
Bookmark advanced on a run that failed during the write	No error visible. A window is missing and the bookkeeping says it succeeded.	Gapped: the position was committed before the data was durable.	Roll the bookmark back to the last verified window and re-run forward. This is safe only if the write is keyed and idempotent, which is why that property is not optional (Idempotent Data Pipelines).
Catch-up launched at full concurrency after a six-hour outage	The source slows, then fails, then the pipeline's own retries make it worse.	Recovery load exceeding what a just-recovered system can absorb.	Cap concurrency, add jitter, and accept a longer recovery. Prefer one slow recovery to two fast outages (Retry Storms: The Load You Generated Yourself).

Catching up without causing the next incident

TOOL-SPECIFICOrchestrators differ on whether the missed-window queue is created for you: a system with logical-date semantics and catch-up enabled will build the queue automatically and, without a concurrency cap, launch it all at once — while a plain cron schedule never builds the queue at all, so step 3 is a manual enumeration.

Once the classification says "late" or "gapped", the work is a queue of units — missed windows, or offset ranges not yet landed — and the only real question is how fast to drain it. The instinct is to drain it as fast as possible, and that instinct causes a large fraction of second incidents.

A pipeline that was down for six hours has six hours of work waiting, and it will attempt it against systems that have just recovered and may still be fragile. Six hours of extracts launched simultaneously against a database is a load pattern that database has never seen, including at its own peak. The same is true of API quotas, which are usually sized for steady state.

The runbook below is deliberately boring, and its order matters. Classification comes before action because the actions are mutually exclusive. Capping concurrency comes before starting because it cannot be retrofitted once the queue is draining. Reprocessing downstream comes last because it is pointless until raw is correct, and it is the step most often forgotten because the ingestion dashboards go green before it happens.

How aggressively should catch-up run?

What is the binding constraint — the retention clock, the source's capacity, or the consumer's patience?

Full speed, all units at once

when Almost never. Only when the source is known-idle and dedicated to this — a file archive, a replica nobody else reads.

cost Load spikes far above steady state on systems that may still be fragile. Buys the shortest stale period and risks a second outage (Retry Storms: The Load You Generated Yourself).

Capped concurrency with jitter

when The default. The source is shared, and recovery over a longer period is acceptable.

cost A longer stale window for consumers, in exchange for bounded load and a recovery that does not become the incident (Without Jitter, Every Client That Failed Together Retries Together).

Newest first, then backwards

when Consumers need current data urgently and historical repair can follow.

cost The table has a moving hole in it while recovery proceeds, which is the most dangerous state to serve unless the gap is published. Buys current correctness first (Stale Dashboards).

Oldest first, in order

when Downstream models process windows in sequence, or a "latest state" model would be corrupted by out-of-order arrival.

cost Current data stays stale until the backlog clears, which for a long outage is a long time. Buys ordering that downstream may depend on (CDC Ordering and Transaction Boundaries).

Pause and escalate before draining

when Lag is close to the retention boundary, or the volume of missing data is large enough that recovery itself is a project.

cost Time, which is the resource in shortest supply. Buys a decision made deliberately rather than a queue drained into a source that cannot take it.

INGESTION RECOVERY RUNBOOK

1. CLASSIFY                       late | gapped | lost
   |
   +- what is the retention boundary for this source?
   +- how long until late becomes lost?          <- the clock for this incident
   +- which datasets and which time range are affected?

2. STOP THE BLEEDING
   |
   +- if the source is degraded: pause ingestion, do not retry into it
   +- if a poison record blocks a partition: route it to dead letter, resume
   +- if the bookmark advanced past unlanded data: roll it back NOW,
      before anything else advances it further

3. ENUMERATE THE UNITS
   |
   +- batch     -> list of missing windows        [w1, w2, ... wn]
   +- streaming -> offset range per partition     [p0: a-b, p1: c-d, ...]
   +- if the units cannot be enumerated, that is the finding of the incident

4. DRAIN, SLOWLY
   |
   +- cap concurrency per source (not per job)
   +- jitter start times; never launch n windows at t=0
   +- watch: source latency, error rate, catch-up throughput vs production rate
   +- if source health degrades -> reduce concurrency, do not retry harder

5. VERIFY BEFORE PUBLISHING
   |
   +- continuity:    every unit in step 3 now present
   +- uniqueness:    source key unique across the recovered range
   +- reconciliation: source vs raw counts for the affected closed period
   +- only then publish atomically into what consumers read

6. REPROCESS DOWNSTREAM
   |
   +- walk lineage forward from raw: staging -> models -> marts -> extracts
   +- re-run each for the affected range only, in dependency order
   +- ingestion green does NOT mean the dashboards are correct

7. RECORD IT
   |
   +- affected datasets, time range, magnitude, cause, whether recovered
   +- annotate freshness and volume charts for that range
   +- an undocumented gap is a future incident with a longer investigation

How deep does recovery actually go

Recovery depth is not a property of your pipeline. It is a property of what every system upstream of the damage still holds, and it is worth writing down per source before an incident rather than discovering it during one.

The chain below is the practical form of that question. Each node holds something for a bounded period, and each is capable of corrupting the recovery in a specific way — which is the part usually left out. A source that still has the rows can also have changed them. A broker that still has the events can also have compacted them by key, so intermediate changes are gone while the latest one remains.

Read the chain from the top during an incident. The first node that still holds the affected range intact is where recovery starts, and everything below it must be reprocessed forward from there. The first node that does *not* hold it tells you what kind of incident you are in.

Recovery sources, deepest first, and how each one can mislead

Source system, current state
holds Every row that still exists, with today's values.
could corrupt Re-extracting a historical range returns current values, silently rewriting the affected period with a state that never existed then. Deleted rows are absent entirely (What Backfills Break).
↑ reads from
Source change log (WAL, binlog)
holds Every committed change in commit order, for the server's retention.
could corrupt Nothing, while it holds the range — this is the highest-fidelity recovery available. But retention is set by a DBA for the source's needs, not for yours, and it is usually short (Write-Ahead Logging).
↑ reads from
Broker topic
holds Every published event, per partition, for the configured retention.
could corrupt A compacted topic keeps only the latest event per key, so replay reconstructs state and silently loses transitions. Retention expiry removes the range with no marker (Retention and Replay).
↑ reads from
Dead-letter destination
holds Records that failed to decode, with their raw bytes and positions.
could corrupt It is frequently the only copy of those records, and it is frequently unmonitored and unretained. Replaying from it out of order can produce stale current state (A Dead-Letter Queue Is a Workflow, Not a Bin).
↑ reads from
Raw landing zone
holds Everything that successfully landed, exactly as it arrived, indefinitely.
could corrupt Nothing, if it is genuinely immutable. If anything ever cleans, dedupes or rewrites in place, this node is not a recovery source at all — it is just another derived table (The Raw Landing Zone).
↑ reads from
Staging models
holds Cleaned, typed, deduplicated rows derived from raw.
could corrupt Recovering from here rather than from raw bakes in whatever the cleaning logic was at the time, including the bug you are recovering from.
↑ reads from
Serving tables and marts
holds Modelled, aggregated output that consumers read.
could corrupt Not a recovery source in any circumstance, though it is regularly used as one during incidents because it is the most convenient thing to hand (Reprocessing vs Retrying).

Recovery starts at the highest node that still holds the affected range intact, and everything below it is reprocessed forward. The reason the raw landing zone must be immutable is visible here: it is the only node in the chain whose retention you control and whose contents nothing else can corrupt.

Verifying that a recovery actually recovered

Check	Expresses	Catches	Still misses
Unit continuity across the affected range	Every window or offset range that was missing is now present.	A resumption mistaken for a recovery; a catch-up that stopped partway; units nobody enumerated.	A unit that is present but was assembled from the wrong range, which looks identical from the outside.
Reconciliation against the source for the affected period specifically	The recovered period matches the source.	Residual gaps, over-recovery, and a backfill that quietly excluded a slice.	Rows deleted at the source since the incident — they are absent from both sides and reconcile perfectly (Missing Rows).
Uniqueness on the source key across the recovered range	Catch-up did not double-land anything.	A wide re-read appended rather than merged; overlapping recovery runs; a replay into an appending sink.	Duplicates that straddle the recovery boundary, where one copy is inside the range and one outside it.
Downstream freshness and row counts for the affected range, per dependent model	The repair propagated all the way to what consumers read.	The commonest post-incident defect: raw repaired, models not reprocessed, dashboards still wrong.	A model whose logic was itself wrong during the period, which reprocessing faithfully reproduces (The Pipeline Succeeded. The Data Is Wrong.).

The last row is the one to build first. Ingestion recovering is a milestone; consumers seeing correct numbers is the actual goal, and the two are separated by every transformation in between.

How to build it

Most important first.

Make the failure unit addressable before you need it: explicit window parameters for batch, per-partition offset tracking for streaming, and a run log that records which units completed. Recovery is easy or impossible depending on decisions made months earlier (The High-Water Mark). Derive the next unit from the last completed one rather than from the clock, so an outage produces a queue of pending units rather than a silent gap — this single convention turns most gaps into automatic catch-up (Batch Ingestion).
Bound catch-up: cap concurrency, add jitter, and prefer slow full recovery over fast partial recovery. Recovering over an hour without a second incident beats recovering in ten minutes and taking the source down (Rate Limiting).
Alert on lag as a fraction of the retention that bounds recovery, not as an absolute duration. The alert should tell the on-call engineer how much time they have left, which is the only number that matters during the incident (Depth Is Not an Emergency; Age Is).
Route undecodable records to a dead-letter destination with the raw bytes and the position, and alert on any arrival. Skipping loses the record; blocking loses everything behind it; a dead letter loses neither and defers the decision (A Dead-Letter Queue Is a Workflow, Not a Bin).
Reprocess downstream from the earliest affected stage, not from the beginning. A lineage graph is what makes that possible to determine quickly (Data Lineage). Validate the repair before publishing it, because recovery is the moment when a wrong write is most likely and least scrutinised (Validating a Backfill Before You Publish).
Record affected periods as data — a table of incidents with their date ranges and affected datasets — so future analysts investigating an anomaly can distinguish an incident from a business event (Data Incidents).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A re-run guarantees the same result as the original run only if the extract is a deterministic function of its parameters. Any extract reading now(), joining a mutable dimension, or asking a source for "recent changes" is not, and its re-run is a different operation with a similar name.
Catch-up guarantees ordering of processing, not ordering of data. Windows processed out of order can leave a "latest state" model holding an older value unless latest is chosen by a source column (CDC Ordering and Transaction Boundaries).
Replay guarantees delivery of everything still within retention, at least once. It guarantees nothing about what expired, and the boundary is not marked in the data.
An idempotent keyed write guarantees that a re-run converges to the same state regardless of how many times it runs. That is the only guarantee in this lesson that makes recovery safe rather than merely possible (Upserts and Merges).
Nothing guarantees that a recovered period equals what would have been ingested at the time. Re-extracting historical rows from a mutable source returns their current values, which is a different dataset with the same row count (What Backfills Break).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

Continuity assertion over the recovery unit — every expected window present, every offset range covered per partition. It is the check that distinguishes "resumed" from "recovered" and it is the one most often absent.
Post-recovery reconciliation against the source for the affected period specifically, not for a rolling window. The rolling window will pass again as soon as the incident scrolls out of it, which is a way of forgetting rather than of verifying (Reconciliation).
A uniqueness assertion on the source key across the recovered range, because the most common recovery defect is duplication rather than omission — catch-up ran wide and the write was not keyed.
What these still miss: a recovered period assembled from a source that has changed since. Counts match, keys match, and the values are today's rather than the period's. Only a note attached to the dataset communicates that, because no check can detect it (Dataset Documentation).

Freshness

During recovery, freshness is dominated by catch-up throughput rather than by the ingestion design. A capped catch-up trades a longer stale period for a lower risk of a second outage, and that trade should be made deliberately rather than by whatever concurrency the code happened to have.
Consumers experience two distinct states and should be told which one they are in: stale-but-correct, where data is old and complete as far as it goes, versus fresh-but-incomplete, where recent data is present and an older window is still missing. The second is far more dangerous because it looks healthy (Stale Dashboards).
A partially recovered dataset with a hole in the middle is the worst state to serve, because freshness monitoring shows green — the newest record is recent — while a period in the middle is wrong (Freshness Monitoring).
Where a freshness SLO exists, recovery is measured against it rather than against feelings. Where it does not, the incident will end when someone stops asking (The Freshness SLO).

When the schema or meaning changes

Recovery frequently spans a schema change, so the code re-running an old window is not the code that originally ran it. Whether that is a fix or a corruption depends on which behaviour the downstream models were built against (Schema Evolution).
A connector upgraded during the incident may bookmark differently — a different position format, a different commit ordering — and the seam between old and new positions is a place a gap can hide (Snapshot and Stream: the Bootstrap Problem).
Adding idempotency during an incident changes the duplicate profile of everything after the change, which is safe only if downstream already deduplicates. Introducing both at once during recovery is how a repair becomes an incident.
The incident log itself is a dataset with a schema, and it will be queried during the next incident. Treat it as a product rather than as a wiki page (Data Incidents).

How to re-run this safely

Classify first: late, gapped or lost. Late needs patience and capped catch-up. Gapped needs an explicit re-request of the missed units. Lost needs a decision about what to tell consumers, because there is no technical action remaining (Missing Rows).
Re-request by unit — window or offset range — never by "restart from the beginning" and never by "resume from now". Both extremes are available by default and both are wrong.
Repair into a location consumers are not reading, validate, then publish atomically. A recovery that overwrites the live table while people query it turns one incident into two (Atomic Publish).
Reprocess downstream from the earliest affected stage forward, using lineage to determine what that is. Re-running everything is slow and re-running too little leaves the dashboards wrong (Reprocessing vs Retrying).
Where data is genuinely lost, record it: the range, the datasets, the estimated magnitude and the reason. A documented gap is a known limitation; an undocumented one is a future incident where somebody spends a week rediscovering it.

What can go wrong

Failure modes

Resuming from the present, leaving a permanent hole in the middle of an otherwise complete table.
Catch-up load taking down the source that had just recovered.
A streaming consumer restarting past its expired offset and silently starting at the oldest available position.
A re-run appending rather than replacing, duplicating the affected period.
Ingestion repaired while downstream models are not reprocessed, so every serving table still shows the gap and ingestion metrics say everything is fine.
The mitigation fails: an automatic retry policy that hides a source outage until the retries exhaust hours later, so the incident is detected at the end of the retry budget rather than at the start of the outage (Retries in Pipelines) — or a dead-letter queue accumulating messages nobody consumes, so the pipeline is green while a growing fraction of the data is parked.

Misreads

"The pipeline is green, so we recovered." Green means it is running now. Whether the affected period was repaired is a separate question with a separate check, and conflating them is how gaps become permanent (The Pipeline Succeeded. The Data Is Wrong.).
"We can always replay." Only within retention, and only into a sink that tolerates redelivery. Both conditions fail routinely and neither failure is announced.
"Backfilling the gap fixes it." It fills it with the source's current view of those rows, which for anything mutable is not what would have been ingested at the time. Sometimes that is fine; it is never automatically fine (What Backfills Break).
"Recovery is an ingestion problem." Ingestion is where it starts. Every model, mart and dashboard downstream of the affected range is also wrong until reprocessed, and ingestion metrics will not tell you that (Data Lineage).
"Retry harder." Retries against a source that is down are load against a source that is down. Backoff, jitter and a circuit breaker recover faster than aggression does, because they let the source recover first (Circuit Breaker).
"The gap is small enough to ignore." Perhaps — but that is a decision for the consumer of the metric, made with the range and the magnitude in front of them, not a decision for whoever was on call.

Privacy, retention and access

A recovery that re-extracts historical rows may re-ingest personal data that was deleted at the source in the interim, silently undoing a deletion request. Any backfill crossing a deletion needs the deletion list applied afterwards (Deletion Requests).
Dead-letter queues accumulate raw payloads indefinitely by default, frequently including personal data, in a location outside the retention and access controls applied to the main pipeline (PII in Pipelines).

Operating it

How you see it in production

Pending units of work — missed windows not yet caught up, or offset ranges not yet landed — as a queue depth. It is the number that says whether recovery is progressing, and it is different from "the job is running" (The Backlog Arithmetic: Four Levers and a Drain Time).
Lag against the retention that bounds recovery, expressed as time remaining rather than time elapsed. This converts a monitoring number into a decision (Depth Is Not an Emergency; Age Is).
Catch-up throughput versus production rate. If the first is not comfortably above the second, the pipeline is not recovering, it is merely running (Little's Law as Working Intuition).
An incident annotation on every freshness and volume chart, so anomalies in history can be attributed rather than re-investigated ("What Changed?" — Deploy Markers and the Invisible Deploys).
Dead-letter arrival rate and dead-letter queue depth, alerted on any non-zero value.

What changes at 10x and 100x

At 10x volume, catch-up after an outage of the same duration takes 10x the work, so the outage duration a platform can absorb shrinks even though nothing about the pipeline changed.
At 100x, catch-up must be throttled or it becomes the incident. Concurrency caps and jitter stop being politeness and become the mechanism that makes recovery possible at all (Backpressure).
With many sources, incidents overlap and the recovery queue is shared. A per-source cap prevents one source's catch-up from starving another's, and its absence is why one connector's outage delays twenty.
What does not change with scale: classification. Late, gapped or lost is the same first question at any volume, and answering it wrongly costs the same amount of wasted time.

What drives cost here

Catch-up concentrates a period of normal cost into a short window, which is why it hits rate limits and capacity ceilings that steady-state operation never approaches.
Reprocessing downstream is usually the larger cost of the two. Re-running transformations over the affected range costs compute proportional to the range times the number of dependent models (Compute Waste).
Idempotent keyed writes cost more per row than appends, every day, in exchange for recovery being safe. It is an insurance premium and the claim is made during incidents (Upserts and Merges).
Longer retention on brokers and source logs costs storage continuously and buys a larger recovery window. That is the trade to argue explicitly rather than accepting whatever the default was (Storage Lifecycle).

What this approach costs

Capped catch-up buys protection against a second outage and costs a longer stale period. Consumers usually prefer the slower option once the alternative is explained, and are rarely asked.
Idempotent keyed writes buy safe re-runs and cost merge overhead on every ordinary run forever.
Longer retention buys a deeper recovery window and costs storage continuously — an insurance policy whose premium is visible and whose payout is not.
Automatic retries buy resilience to transient failures and cost detection latency for persistent ones. An aggressive retry policy can hide a source outage for hours, and the alert should fire on the first failure rather than at the end of the budget (Retries).
Recording incidents as data buys future analysts the ability to attribute anomalies and costs the discipline of maintaining it during exactly the moments when nobody wants to write anything down.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe late/gapped/lost classification and the rule that recovery depth is bounded by somebody else's retention hold for every ingestion design. What changes is which boundary applies — broker retention, source log retention, or an API's change-history window.
BROKER-SPECIFICA Kafka consumer whose committed offset has aged out resets according to its auto-offset-reset policy — to the earliest or latest available — and the two produce opposite failures: a silent replay of everything, or a silent skip to the present. Pub/Sub, which acknowledges individual messages rather than tracking an offset, expires unacknowledged messages instead and fails differently.
TOOL-SPECIFICWhether missed schedule intervals are automatically queued or silently dropped is an orchestrator behaviour, not a property of your code: an Airflow-style catch-up will attempt every missed interval at once unless concurrency is capped, while a plain cron entry simply never revisits them.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasecrash-recovery write-ahead-logging replication

Backendretries backoff-and-jitter dead-letter-queues circuit-breaker job-idempotency retry-storms graceful-shutdown queue-backlog

Architecturecircuit-breaker reliability-patterns backpressure idempotency

Observabilityqueue-age queue-backlog littles-law deployment-markers incident-debugging incident-timeline

Clouddisaster-recovery rpo-and-rto backup-strategy storage-lifecycle

Domains that do not exist yet

— DevOps / Production Engineering owns the incident process around all of this — paging, severity, comms, and the retrospective that turns one recovery into a scheduling change.
— Distributed Systems owns why a position commit and a durable write cannot be made atomic across two systems without a protocol, which is the underlying reason the ordering of those two operations decides whether a crash loses data or duplicates it.