Pipeline Reliability
Reliability is not a low failure rate. It is seven mechanisms — retries, idempotency, checkpoints, atomic publish, validation, rollback, reprocessing — that are only safe as a set.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The nightly pipeline ran 340 times last month and failed twice. Is it reliable?
Everyone who reads a serving dataset without being able to see how it was built: the analyst who assumes yesterday is complete, the finance team closing a month, the scheduled export that fires at 07:00 regardless, and the downstream model whose own run treats your table as an input contract (Who Actually Consumes This Data). None of them can distinguish "this table is correct" from "this table exists".
Reliability is defined over the publishable unit — the smallest chunk of output that can be produced, validated and made visible as a whole. Usually a partition-day, sometimes a whole table, sometimes one file. Choosing that unit is the first reliability decision anyone makes, and choosing it badly is why a two-partition failure turns into a full rebuild (Partial Failure).
Wrap every task in three retries, alert when the DAG goes red, and page whoever is on call. This is the reliability model every orchestrator ships with, it is genuinely better than nothing, and it converts the large class of transient failures — a network blip, a warehouse restart, a token that expired — into a self-healing non-event. Most pipelines start here and many stay here for years without visible harm.
A task times out *after* it wrote its rows. The retry runs the same insert again and the fact table now double counts the day, while the orchestrator reports two attempts and one success (Duplicate Rows).
- A task times out *after* it wrote its rows. The retry runs the same insert again and the fact table now double counts the day, while the orchestrator reports two attempts and one success (Duplicate Rows).
- The transform writes to
fct_ordersin fourteen statements and dies after the ninth. Consumers querying at that moment read a table that never existed as a consistent whole, and there is no marker anywhere saying so (Atomic Publish). - The streaming job restarts, replays from its last committed offset, and re-emits four minutes of aggregates it had already written — because the offset commit and the state snapshot were saved separately and the crash landed between them (Checkpointing).
- A hundred partitions are processed; ninety-eight succeed. The orchestrator marks the *task* failed, so the operator re-runs the task, so ninety-eight correct partitions are computed a second time and one of them is not idempotent (Partial Failure).
- Someone reverts the transformation commit after a bad deploy and reports the incident resolved. The code is back; the wrong rows the bad code wrote are still sitting in the warehouse, and will be there next quarter (Rolling Back Data).
- The upstream extract silently returned zero rows. Every task succeeded, the run was faster than usual, and nothing in the reliability model has any opinion about the number of rows a successful run should produce (The Pipeline Succeeded. The Data Is Wrong.).
What is actually happening
- A pipeline run is not a request. A request either affects the world or does not, and the caller learns which. A pipeline run leaves durable, visible, shared state behind — partitions, files, table versions — and it can leave that state half-formed, with no caller anywhere to be told.
- That is why the seven mechanisms exist and why they are mutually dependent. Retry makes transient failure survivable but is only safe if the task is idempotent. Idempotency is only affordable if there is a defined unit to be idempotent *over*, which is what atomic publish gives you. Atomic publish is only worth doing if something validates before the swap. Validation is only actionable if you can roll back or reprocess. And reprocessing is only possible if a checkpoint or an immutable raw copy tells you where to restart from (Keeping Raw History: The Recovery Position and the Liability).
- Take any one away and the neighbours become unsafe rather than merely weaker. Retries without idempotency actively manufacture duplicates. Atomic publish without validation publishes wrong data atomically, which is worse than publishing it visibly in pieces because nobody notices. Validation without rollback produces an alert and no remedy.
- The orchestrator sees exactly one of the seven. It knows whether a process exited zero. It does not know what that process wrote, whether it wrote it twice, whether the write was complete, or whether the numbers are plausible — and no amount of task-level retry configuration teaches it (When a Task Fails Mid-DAG).
- This is why the reliability question in this domain is phrased over *outputs* and not over *runs*: for every publishable unit, is it present, complete, correct-once, fresh enough, and reversible? A run is a means of producing that; its exit code is a weak proxy for whether it did.
Seven mechanisms, and why none of them works alone
The mechanisms below are usually taught as a checklist, which is exactly the wrong shape. They are not seven independent improvements you can adopt in any order; they are a set with hard dependencies, and adopting one of them in isolation frequently makes the platform worse than adopting none.
Read the without it column as the operative one. It is the reason a team that turned on retries last quarter is now debugging duplicate revenue, and the reason a team that built a beautiful atomic publish path still shipped a wrong number to the board.
The dependency structure also tells you the adoption order. Idempotency first, because it is the precondition for retries and re-runs alike. Then a defined publishable unit and an atomic swap. Then validation in front of the swap. Rollback and reprocessing last, because they are the mechanisms you reach for after the first three have told you something is wrong.
| Mechanism | What it buys | What it depends on | Without it |
|---|---|---|---|
| Retries | Transient failures — a dropped connection, an expired token, a restarted warehouse — stop being incidents. | Idempotency of the task over the publishable unit. | Every transient failure requires a human, and on-call becomes a re-run button with a pager attached. |
| Idempotency | Running the same unit twice leaves the same result as running it once, so retries and re-runs are both safe. | A stable business key and a defined unit to replace. | Retries manufacture duplicates and backfills double count. This is the load-bearing mechanism of the set. |
| Checkpointing | A restarted job resumes from a known position instead of from the beginning or from nothing. | Input position and state committed together, in one atomic action. | A restart either loses work or repeats it, and stateful streaming becomes unrunnable at any real scale. |
| Atomic publish | No consumer ever observes a half-written dataset. | The ability to build somewhere consumers are not reading, and one operation that flips visibility. | Readers see partial data as though it were complete, which reads as a quiet business day rather than as a fault. |
| Validation | Wrong output is caught before it becomes visible rather than after. | A gap between build and publish in which to run the assertions. | The publish path is a straight line from a bug to a dashboard, and detection is delegated to whoever reads it first. |
| Rollback | A bad publish can be undone as an operation instead of an investigation. | A retained previous version, and knowing which units the bad run touched. | Incidents are resolved by reverting code while the wrong rows stay in the tables indefinitely. |
| Reprocessing | History can be corrected once the logic is fixed. | Immutable inputs, deterministic transformations, and a bounded range. | Every bug discovered after the fact is permanent, and the platform accumulates periods that are known-wrong and cannot be repaired. |
The dependency chain reads right to left: reprocessing needs immutable inputs, rollback needs versions, validation needs a gap before publish, publish needs a unit, and everything needs idempotency. That is the adoption order, and skipping to the middle of it is the most common way a platform ends up with sophisticated tooling and duplicated revenue.
The green run and the wrong table
Two teams operate the same pipeline. Both report to their leadership that it succeeded every night last month. One of them has been publishing a fact table that has double-counted every retried partition since a timeout policy changed in March; the other has not. Nothing on either orchestrator distinguishes them.
The distinction is not effort or diligence. It is *where the reliability question is asked*. Asking it about the run gets you exit codes and durations. Asking it about the output gets you presence, completeness, uniqueness and freshness — properties of data, which is the thing consumers actually depend on.
This reframing has a practical consequence that is easy to miss: it changes what an on-call engineer does at three in the morning. Under run-oriented reliability, the correct response to a failure is to re-run the task. Under output-oriented reliability, the first question is which publishable units are affected, and re-running blindly is a specific and common way of turning one bad partition into thirty.
Instrument the orchestrator. Alert on task failure and on runs exceeding their duration budget. Configure three retries with exponential backoff on every task uniformly. Report monthly on the percentage of successful runs, and treat a green month as evidence the platform is healthy.
Instrument the outputs. For every publishable unit record whether it was published, when, by which run, and how many times. Assert completeness and uniqueness before the swap that makes it visible. Configure retries per task according to whether that task is idempotent. Report on units that are late, missing, republished, or that failed validation — and treat a green month with no reconciliation as an unmeasured month.
Task status can only report failures that raise an exception. Duplicated rows, missing partitions, half-written tables and stale-but-present data all produce zero exit codes by construction, so a monitor watching runs is structurally blind to the entire failure class this domain exists to address. Moving the instrumentation to the output does not require better tooling — it requires recording what was published rather than what was executed.
How one transient failure becomes a permanent wrong number
The most instructive incident in this module is also the most boring one. A warehouse connection drops during a nightly load. Nothing exotic happens. The mechanisms that were supposed to contain it interact, and a temporary network problem is converted into a permanently wrong revenue figure that survives until someone reconciles a quarter.
Follow the path below and notice how many times the system is behaving exactly as configured. There is no bug in the orchestrator, no bug in the warehouse and no bug in the transformation SQL. The failure is entirely in the composition — a retry policy applied to a task that publishes non-idempotently, with no validation between the write and its visibility.
That is the shape of nearly every reliability incident in data platforms: correct components, composed without asking what each one promises the next. It is also why the response "add more retries" makes it worse, and why the response "remove the retries" makes the pipeline fragile without making it correct.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Connection drops after commit, before acknowledgement. | Task fails despite its write having succeeded. | The write and the report of the write are not the same event, and no protocol makes them one. | Accept that this is unavoidable and design the retry to be safe rather than trying to make the failure impossible. |
Retry re-runs a bare INSERT. | Every row for the partition exists twice. | The task appends rather than replacing a defined unit, so running it twice means two of everything. | Replace the append with a delete-then-insert inside one transaction, or a merge on the business key. Costs a scan of the partition per run. |
| Rows are written directly into the table consumers query. | A scheduled export at 02:15 captures a state that is neither the old day nor the new one. | No separation between building the output and making it visible. | Build into a staging location and swap. Costs one extra write and delays visibility (Atomic Publish). |
| Nothing asserts row counts before publish. | Doubled data is visible and looks like an excellent day. | Validation, where it exists at all, runs on a schedule after the fact rather than in the publish path. | Put a uniqueness assertion on the business key between build and swap. Costs one aggregate query and blocks the publish when it fires. |
| Incident is closed by re-running the task. | A third copy of the partition. | The runbook is written in terms of tasks, not in terms of published units. | Write runbooks that name the affected units and the repair operation, never "re-run the DAG" (Data Incidents). |
| The fix itself is deployed without a repair. | New days are correct; the affected day stays wrong forever. | Fixing forward and repairing history are two separate pieces of work and only the first one feels urgent. | Treat every incident as producing two tickets — stop the bleeding, and repair the range (Reprocessing vs Retrying). |
How to build it
Most important first.
- Choose the publishable unit explicitly and write it down — partition-day, table, file set. Every other mechanism in this module is defined relative to it, and teams that never chose one end up with mechanisms defined over different units that do not compose.
- Make every task idempotent over that unit before turning on any retry.
DELETE WHERE partition = :d; INSERT ...inside one transaction, or aMERGEon the business key, is the entire technique in most warehouses (Upserts and Merges, Idempotent Data Pipelines). - Publish atomically: build somewhere consumers are not reading, validate there, then make it visible in one operation (Atomic Publish).
- Run the validation between the build and the swap, not after it. A test that runs after publish is an incident detector; a test that runs before publish is a reliability mechanism (Data Tests).
- Keep the previous version reachable — a table snapshot, a retained prior partition, an immutable raw layer — so that "undo" is an operation rather than an archaeology project (Rolling Back Data).
- Give every stateful job a checkpoint that commits its input position and its state together, and give every stateless one a defined restart point derived from data rather than from a clock (Checkpointing, The High-Water Mark).
- State the promise as an SLO on the dataset, not on the DAG. "
fct_ordersis complete for day D by 06:00" is checkable, ownable and arguable; "the pipeline is reliable" is none of those (Pipeline SLOs).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A green orchestrator run guarantees that a process exited zero within its timeout. It says nothing about rows written, rows missing, rows duplicated, or whether the output is the same as it would have been yesterday.
- Retries guarantee that transient failures do not become permanent ones. They guarantee nothing about how many times an effect occurred, and by construction they make repeated effects *more* likely, not less.
- Atomic publish guarantees that no reader observes a partially built dataset. It does not guarantee that the dataset is correct, complete, or built from complete inputs.
- Validation guarantees that the assertions you wrote hold on the data you tested. Every property nobody encoded is unconstrained, which is most of them (Data Quality).
- Nothing anywhere in this list guarantees completeness against the source. Completeness is measured by reconciliation, never received (Reconciliation).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that tests the whole model at once: for a closed period, assert that the publishable unit exists, that its row count matches the source within a stated tolerance, that the business key is unique, and that the unit was published exactly once. Four assertions, run before the swap.
- It misses everything about periods that are still open, which is where late data lives (Late-Arriving Data), and it misses any error that is present identically in source and target.
- The subtler miss: it validates the data, not the mechanism. A pipeline whose retries are unsafe passes every one of those assertions on every run where nothing was retried, which is almost every run — so the failure is rehearsed only in production, at the worst moment.
- Reliability mechanisms cost latency, and they cost it in the direction consumers feel: build to a staging location, validate, then swap means the data becomes visible later than a direct write would have made it visible.
- Retries with backoff extend the tail of a run, not its median. The freshness figure a consumer should be given is the one that holds when a retry happens, because the day a retry happens is the day someone notices.
- The honest framing is a trade between freshness and the probability that what is visible is wrong. Publishing early and correcting later is a legitimate choice for some consumers and unacceptable for a monthly close; the mistake is making that choice implicitly (Cost vs Freshness).
- When the publishable unit changes — from daily partitions to hourly, from full table to incremental — every mechanism defined over the old unit needs revisiting at the same time, and this is almost never done in one change. The usual outcome is idempotency defined per day and publishing defined per hour, which is not idempotent at all.
- When a schema change adds a column, the reliability question is whether an old-shaped and a new-shaped partition can coexist in one table without breaking a reader mid-migration (Schema Evolution).
- When the *meaning* of a measure changes, no reliability mechanism helps: the pipeline reliably publishes a number that means something different from what it meant last week (Semantic Changes).
- Recovery from a bad run is bounded by the earliest thing you still have that is known good — the previous table version, the retained partition, the raw landing zone, or the log you can replay (Replay from the Log).
- Recovering *forward* (recompute the affected range and republish) is usually preferable to recovering *backward* (restore an old snapshot), because forward recovery leaves you with correct data and backward recovery leaves you with old data (Planning a Backfill).
- The failure mode of recovery itself is scope: a repair run that recomputes more than the affected range, over a source that has since changed, produces a second incident whose blast radius is larger than the first (What Backfills Break).
What can go wrong
- Retries that duplicate rows because the task published before it failed.
- Idempotency implemented on the wrong key, so a genuine second event and a redelivery are indistinguishable (Deduplication).
- Atomic publish implemented as a directory rename on object storage, where a rename is a copy plus a delete and is not atomic at all.
- Validation that runs after publish, so consumers read the bad data during the interval between publish and alert.
- A checkpoint that stores the input position and the state separately, so a crash between the two writes reintroduces the dual-write problem the checkpoint existed to avoid (The Dual Write Problem).
- A rollback that reverts code and not data, closing the incident with the wrong numbers still in place.
- The mitigation failing quietly: an alert routed to a channel nobody reads, a test disabled six months ago "temporarily", a retry policy set to zero on the one task that needed it (Alert Fatigue: The Page Nobody Reads).
- "Our pipelines are reliable — we are at 99.9% successful runs." Success rate measures the mechanism you can already see. The failures that cost the most are inside the successes (The Pipeline Succeeded. The Data Is Wrong.).
- "Retries make the pipeline reliable." Retries make transient failures survivable and permanent-effect failures more frequent. Which of the two you get is decided entirely by idempotency (Retries in Pipelines).
- "We validate the data, so we are covered." Validation placed after publish detects; it does not protect. The position of the test in the pipeline matters as much as the test.
- "Reliability is an infrastructure concern." Almost every failure in this module is a *design* choice about units, keys and publish order. Better infrastructure does not fix a non-idempotent merge.
- "We have never had a data incident." More often this means nobody has checked. Platforms without reconciliation do not have fewer incidents; they have undiscovered ones.
Operating it
- Per publishable unit, not per task: was it published, when, by which run, and how many times. A publish ledger is the single highest-value table a data platform can keep about itself (Pipeline Metrics).
- Retry count per task per run, trended. A task whose retries are rising is telling you about an upstream degradation long before it fails outright (Pipeline Observability).
- Time from run start to publish, at the tail rather than the average — the SLO is broken by the slow day, not the typical one (Percentiles: Which One, and How Many Users Is That?).
- Validation outcomes recorded as data: which assertion, which unit, pass or fail, retained over time. Tests whose history is not kept cannot tell you whether quality is improving (The Data Quality Dashboard).
- At 10x volume the publishable unit usually has to get smaller, because a full-table swap stops fitting in the window. Smaller units mean more of them, which makes per-unit publish tracking mandatory rather than nice to have.
- At 100x, partial failure stops being an exception and becomes the normal shape of a run: with enough tasks, something always fails, and a model that treats any failure as "re-run the job" is arithmetically doomed (Straggler Tasks).
- Consumer count scales the *rollback* problem rather than the publish problem. One consumer reading a bad table is a fix; eighty dashboards and four downstream models reading it is an impact-analysis exercise (Impact Analysis).
- The build-validate-swap pattern writes the output roughly twice — once to the staging location and once, logically, at publish — plus the read the validation performs. That is the standing price of never showing a consumer a half-built dataset.
- Retries cost repeated compute on exactly the work that was already expensive enough to fail, and an unbounded retry policy against a failing dependency converts one broken run into a sustained load on it (Retry Storms: The Load You Generated Yourself).
- Keeping previous versions reachable costs retained bytes, and it is the cheapest insurance in the platform: a rollback window is storage you pay for continuously against an event you hope never happens (Storage Lifecycle).
- The cost that is never on the invoice is the incident: engineer hours, a re-run of everything downstream, and a period during which nobody trusts any number (Trusting Data).
- Every mechanism here buys safety with latency, storage or complexity. Staging plus validation plus swap delays visibility and doubles writes. Idempotent merges are more expensive than appends. Checkpoints add state to jobs that were pleasantly stateless.
- The mechanisms are also not independently optional, which is the uncomfortable part: adopting retries without idempotency makes the platform *less* reliable than having neither, because it manufactures a failure class that did not previously exist.
- There is a genuine argument for stopping short. A dataset read by two people once a week, rebuilt from immutable raw in twenty minutes, does not need a publish ledger and a rollback window; it needs a re-run. Building the full apparatus everywhere costs more than the incidents it prevents on datasets nobody depends on.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe interdependence of the seven mechanisms is a property of writing durable shared state on a schedule, so it holds for a dbt project, a Spark job, a Flink pipeline and a hand-written Python script alike. What differs is which mechanisms the tooling gives you for free.
- TOOL-SPECIFICOrchestrators differ in what a "task" can express: some can mark individual mapped units failed and retry only those, others retry whole tasks only. That single capability decides whether partial failure is a routine operation or a manual one, and it is worth checking before designing around it.
- SCALE-SPECIFICBelow a few datasets with a handful of readers, a rebuild-from-raw is a complete reliability strategy and the apparatus here is overhead. Above roughly the point where a rebuild no longer fits inside the freshness window, every mechanism in this lesson becomes mandatory rather than optional.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the delivery side of this — how the transformation code is tested, versioned, promoted and rolled back, and what an error budget means for a deployment cadence. That domain is being built separately; the data-specific half is that reverting the artifact does not revert the rows it wrote.
- — Distributed Systems owns why "the write succeeded but the acknowledgement was lost" is not a solvable problem but a permanent condition, and what the available responses to it are. Every retry decision in this module is downstream of that result.