Data Incidents
A dashboard says revenue dropped eighty percent overnight. Seven different causes produce that symptom, and telling them apart is the job.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Revenue on the executive dashboard is down eighty percent since yesterday. Before touching anything, what could produce exactly that symptom?
The executive who saw the number and is about to act on it, and every other consumer of the same models who has not noticed yet. What they need first is not a cause but a *status*: is the number wrong, and should decisions wait (Trusting Data)?
One incident covers one root cause across every dataset it touched, not one alert per affected table. Getting this grain wrong is why incident channels become unreadable: a single stale source produces dozens of downstream alerts and exactly one incident (Impact Analysis).
Treat it as a bug report against the dashboard. Open the BI tool, check the filters, re-run the query, and if the number is still low, re-run the pipeline that feeds it. Sometimes this works, and when it works it works in ten minutes.
Re-running the pipeline is a repair attempt performed before diagnosis. If the cause was an unsafe backfill, re-running it appends a third copy; if the cause was a broken transform, it republishes the same wrong number with a fresher timestamp (What Backfills Break).
- Re-running the pipeline is a repair attempt performed before diagnosis. If the cause was an unsafe backfill, re-running it appends a third copy; if the cause was a broken transform, it republishes the same wrong number with a fresher timestamp (What Backfills Break).
- The drop is real. Revenue genuinely fell because a payment provider was down for four hours, and an afternoon of pipeline investigation has delayed the business response by an afternoon.
- The dashboard is one of forty reading the same model. Fixing the tile leaves thirty-nine wrong, and nobody told their owners (Data Lineage).
- Someone "fixes" it by patching the mart directly during the incident. The number is right, the model it derives from is still wrong, and the two now disagree permanently (Data Platform Anti-Patterns).
- The incident is closed when the number looks right again, with no record of which periods were affected. Six weeks later a quarterly report re-reads that range and produces a third number (Validating a Backfill Before You Publish).
What is actually happening
- A data incident is a divergence between what a dataset says and what is true, discovered by a consumer. The technical work is diagnosis; the discipline is that the symptom is a value, so the evidence must be values — not logs, not CPU, not task status (From Symptom to Root Cause).
- The candidate causes are a short and stable list, because there are only so many ways a number can be wrong. Something did not arrive; something arrived twice; something arrived late; something changed shape; the logic changed; the filter changed; or the world changed. Everything else is a variation on one of those (What Goes Wrong Between Source and Dashboard).
- They are distinguished by *which checks fail together*, not by which one failed. Missing rows fail completeness and reconciliation; duplicates fail uniqueness and reconciliation; a nulled cast fails validity and reconciliation; a broken transform fails reconciliation alone. The failing set is a fingerprint (Debugging a Data Incident).
- One candidate has no technical signature at all: the business genuinely changed. It is first on the list because it is common, because it costs nothing to check, and because every hour spent on pipeline archaeology while the real problem is a payment outage is an hour the company does not get back.
- Incident response has a specific first move that is not diagnosis: stop the wrong number from spreading. Marking the dataset degraded, pausing dependent publishes and telling consumers costs minutes and prevents the second-order damage of decisions taken on it (Quality Alerting).
Revenue is down eighty percent: seven candidates
The symptom is one number on one tile. Every row below produces it, and several produce it while leaving the orchestrator entirely green. The value of the table is not that it lists causes — it is that the response column is different for every row, which is why guessing is expensive.
The first row is deliberately first. A genuine business change is common, costs one query to confirm or eliminate, and is the cause most likely to be discovered after an afternoon of pipeline investigation. Any incident process that does not check reality first is optimising for the interesting explanation over the likely one.
Read the cause column as a set of hypotheses to be tested rather than a list to be reasoned about. The next section is how you test them, and the answer is always evidence from the data itself — the alternative is a debate in a chat channel that lasts longer than the query would have taken.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A payment provider was down for four hours yesterday. | Revenue is down eighty percent; row counts in the source are down by the same proportion. | Nothing is wrong with the pipeline. The world produced less, and the platform faithfully reported it. | Confirm against the source for the affected window, tell the reporter the number is correct, and stop. This is the first hypothesis because it is cheap to test and expensive to test late (Reconciliation). |
| The change-capture connector was down overnight. | Revenue is low; source counts are normal; the serving table is missing orders that exist upstream. | Changes committed during the outage were never emitted. The pipeline processed what it received, perfectly (CDC Failure Modes and the Retention Deadline). | Replay from the retained log if the gap is inside retention; re-snapshot if it is not. Then re-validate the affected periods before publishing (Replay from the Log). |
| A transformation was deployed yesterday. | Every row is present, unique, fresh and well-typed. Only the totals are wrong. | The logic changed — a filter, a join condition, a sign, a status code excluded. The code does exactly what it was told (Two Dashboards, Two Numbers). | Diff the model against its previous version, reproduce the number both ways for one period, then fix forward and backfill the range the deploy covered (Planning a Backfill). |
| The producer changed a field's type or units. | Row counts are normal; the measure column is null, zero, or off by a constant factor. | A cast that produces null rather than an error, or a semantic change — cents to units, gross to net — that no type check can see (Breaking Schema Changes, Semantic Changes). | Enforce the contract at the boundary so the next occurrence fails the run instead of publishing nulls, then backfill from raw, which still holds the original payload (Contract Enforcement). |
| A dimension gained duplicate keys. | Revenue is *up*, often by an odd multiple, and row counts in the fact table are high. | The join fanned out: each fact row matched several dimension rows. Listed under an eighty-percent drop because the same fault produces a drop when the duplicate keys cause an inner join to lose rows instead (Duplicate Rows). | Assert uniqueness on the dimension's business key as a blocking test, and rebuild the fact table for the affected range (Data Tests). |
| A batch of events arrived after the window closed. | Yesterday looks quiet; today looks unusually busy; the two together are about right. | The window closed on arrival time rather than event time, so events that happened inside the period were counted nowhere (Late Events). | Recompute the affected periods with the late data included, and decide explicitly how much lateness the pipeline will allow going forward (Watermarks). |
| A deduplication filter was tightened. | Row counts and revenue both drop sharply; nothing upstream changed. | The dedup key or window was widened, so legitimate distinct records are now being collapsed. It is a correctness fix that over-corrected (Deduplication). | Reproduce both filters over one period, compare what the new one removes, and re-derive the key from the business definition of a distinct event rather than from what happened to work. |
The evidence that separates them
Every candidate above is a hypothesis, and each one predicts a different pattern of check results. That is what makes the diagnosis mechanical rather than intuitive: run the whole suite over the affected period and read which ones failed *together*.
The critical property is that no single check identifies a cause. Reconciliation fails for almost everything, which makes it an excellent detector and a useless discriminator. Completeness plus reconciliation says rows were lost. Uniqueness plus reconciliation says rows were duplicated. Reconciliation alone — with everything else green — says the logic is wrong, and that is the hardest and most common case (The Pipeline Succeeded. The Data Is Wrong.).
This is exactly the structure the in-repo pipeline model is built to demonstrate, and it is why that model asserts that no two of its faults produce the same failing set. A diagnostic exercise where two causes are indistinguishable from the evidence is not an exercise, it is a coin flip.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Completeness — every source id for the period is present | Nothing was lost between the source and the serving table. | Connector outages, closed extract windows, late events that were counted nowhere, an abandoned run. | Duplicates that offset losses, and any period that is not yet closed. It also cannot say *where* the loss happened, only that it did (Missing Rows). |
| Uniqueness — each business key appears once | Each real-world event is represented by one row and no more. | At-least-once redelivery, a non-idempotent re-run, an appending backfill, a fan-out join. | Duplicates carrying new keys — a producer retry with a fresh event id is two distinct events as far as this can tell (Deduplication). |
| Freshness — the newest complete period is recent | The table is not simply holding yesterday. | The abandoned run and the stopped source, and it distinguishes both from every fault that published something wrong. | Everything about a table that is fresh and wrong, which is most of the list above (Freshness Monitoring). |
| Validity — the measure is non-null and numeric | The values survived the journey as values. | A cast that nulled a column after a producer changed a type — the fault that leaves row counts perfect and every measure empty. | A value that is well-typed and wrong. A price in the wrong currency or the wrong unit passes every type check there is (Semantic Changes). |
| Distribution — the shape resembles history | The mix across categories and the total volume look like a normal period. | Skewed inputs, a category that vanished, a partial load that hit one segment, and a published-nothing run. | Any fault that preserves shape while changing every value inside it, and it fires confidently on genuine business change (Volume Anomalies). |
| Reconciliation — serving total equals source total | The two ends of the journey agree for a closed period. | Very nearly everything, including the logic errors that every other check passes. | Anything wrong identically at both ends, and it requires access to the source — which is the usual reason it does not exist (Reconciliation). |
Five narrow checks and one broad one. Reconciliation is the detector; the other five are the discriminators, and the diagnosis lives in which of them failed alongside it.
Running the incident before you understand it
The first ten minutes of a data incident are spent on something other than diagnosis, and getting that right is worth more than getting the diagnosis fast. A wrong number that everybody knows is wrong costs almost nothing; the same number, believed, is spent on hiring plans and forecasts.
So the ordering is: confirm the symptom is real and not a BI-layer artefact, mark the dataset degraded where consumers can see it, notify the blast radius from the lineage graph rather than from memory, and only then diagnose. Repair comes last and touches the earliest wrong hop, never the dashboard (Lineage Debugging).
The step that is skipped in almost every incident is establishing the affected *range*. "What is wrong" and "which periods are wrong" are different questions with different answers, and a repair scoped to today when the fault has been present for three weeks is how a single incident becomes a quarterly-report surprise (Validating a Backfill Before You Publish).
A consumer reports a number that looks wrong. What happens in the first ten minutes?
when Always, and first. Query the serving model directly for the same period and filters.
cost A few minutes. It eliminates the BI layer — a filter, a join or a cached extract added in the tool — which is a genuine and unglamorous cause that no upstream check can see.
when The model reproduces the number. Query the source system for the same closed window.
cost Requires source access, which is exactly the thing many data teams do not have. Where it exists it separates "the world changed" from "we broke it" in one query (Source of Truth).
when The number is confirmed wrong and the cause is not yet known.
cost Public admission of a problem before you can explain it, and stale downstream data if publishes are paused. Both are cheaper than decisions taken on a number you already doubt.
when Downstream models would propagate the fault into places that are harder to correct — a mart, an extract, a training set.
cost Everything downstream goes stale, and staleness is itself an incident for someone. Justified for tier-1 lineage and rarely for exploratory datasets (The Freshness SLO).
when The number is confirmed wrong and consumers know.
cost Requires the check suite to exist and to be runnable over an arbitrary past period, which is a design decision taken long before the incident (Debugging a Data Incident).
when Only when the cause is already established and the affected range is known.
cost Performed earlier than this, repair destroys evidence and can compound the fault — an appending backfill on an incident caused by an appending backfill is a real and repeated event (What Backfills Break).
How to build it
Most important first.
- Check whether the drop is real before investigating the pipeline. Compare against the source system directly for a bounded window — one query, minutes of work, and it eliminates or confirms the single most likely cause (Reconciliation).
- Declare the incident and mark the affected datasets degraded before diagnosing. Consumers acting on a number you already suspect is the expensive part of a data incident, and it is the part that is entirely preventable.
- Diagnose from the failing-check fingerprint rather than from intuition. Run the full check suite over the affected period and read the *set* of failures, which discriminates between causes that any single signal cannot (Data Tests).
- Establish the affected range before repairing. "Which periods are wrong" is a separate question from "what is wrong", it is answered differently, and skipping it produces a repair that fixes today and leaves three weeks broken (Planning a Backfill).
- Repair upstream of the symptom, never at it. Patching a mart during an incident creates a permanent divergence from the model it derives from, and the divergence outlives everyone's memory of why it exists (Model Layering).
- Close the incident with a signal, not just a fix: the check that would have caught this, added and enabled. An incident review whose only output is repaired data has spent the whole cost and taken none of the value (Data Observability).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The absence of a failing check guarantees nothing about the presence of an incident. Every fault in this module's model passes several checks on its way to the dashboard, and one passes all but a single check (The Pipeline Succeeded. The Data Is Wrong.).
- A confirmed reconciliation against the source for a closed period is the strongest available statement that a period is right, and it is still bounded: it says the totals agree, not that any individual row is correct.
- Repairing the current period guarantees nothing about the history. Forward fixes and backfills are different operations with different risks, and conflating them is a standard way to turn one incident into two (Reprocessing vs Retrying).
- Nothing guarantees the incident is over when the number looks right. A number can look right for the wrong reason — two offsetting errors reconcile perfectly.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The single most valuable query during an incident: source total versus serving total for the affected closed period, plus per-hop row counts for the same period. Together they answer "is it real" and "where did it go" in two queries.
- It misses causes where source and serving are wrong in the same way — most obviously a business-logic error that exists in both the extract and the model (Two Dashboards, Two Numbers).
- It also misses anything about the periods that are still open, which is exactly where late-arriving data lives, so an incident diagnosed on an open period can be diagnosed as a loss that later fills in (Late-Arriving Data).
- The clock that matters in a data incident is time-to-notification, not time-to-repair. Consumers can work around a known-bad dataset for hours; they cannot work around one they believe is fine.
- Diagnosis is bounded below by the coarseness of the affected periods. On a daily pipeline, confirming a hypothesis about yesterday takes as long as recomputing yesterday, which is why per-hop counts — cheap, immediate — earn their place (Pipeline Metrics).
- Repair latency is dominated by the range, not by the fix. Correcting one partition is minutes; correcting a quarter is a planned backfill with its own validation and its own risk of making things worse (Backfills).
- The most under-diagnosed cause is a schema change that types cleanly. A field that changed from gross to net, or from cents to units, breaks every metric derived from it and passes every schema check (Semantic Changes).
- Incidents caused by upstream schema change should end in a contract, not only in a fix. The producer did nothing wrong by their own standards, and the second occurrence is guaranteed without an explicit agreement (Data Contracts).
- Every repair changes history. Record what was changed, for which periods, and when, so that a number that differs between two readings has an explanation rather than becoming folklore.
- Repair in three steps that are always in this order: establish the affected range, recompute into a location consumers are not reading, validate against the source, then publish atomically (Atomic Publish).
- Never repair by mutating the serving table in place while consumers read it. The intermediate states are observable and someone will screenshot one (Planning a Backfill).
- If the underlying data is unrecoverable — retention expired, the source overwrote it — the correct outcome is a documented gap, not a plausible estimate. An estimate that enters a warehouse becomes a fact within a quarter (Dataset Documentation).
What can go wrong
- Repairing before diagnosing, which frequently makes the incident worse and always destroys the evidence.
- Fixing the symptom at the dashboard or the mart, leaving the model wrong and creating a permanent divergence.
- Closing the incident when today looks right, without establishing which historical periods are still wrong.
- Forty alerts for one root cause, so the response is spent triaging notifications rather than diagnosing (Impact Analysis).
- Investigating a genuine business change as a pipeline fault — the most common way an incident wastes a day.
- An incident review that ends in a repaired dataset and no new check, guaranteeing the same incident twice.
- "The number is low, so we lost data." A drop is equally consistent with a genuine business change, and checking that first is one query. It is skipped because it feels like not doing the job (Correlation Is Not the Root Cause).
- "Re-run the pipeline and see if it fixes itself." Re-running is a repair, and repairs before diagnosis destroy the evidence and can compound the fault (Idempotent Data Pipelines).
- "The tests passed this morning, so it happened after that." Tests assert what they encode. The fault may have been present for weeks in a dimension nobody wrote a test for.
- "It only affects this dashboard." A dashboard is a view of a model, and a model has consumers you have not met. The lineage graph is the authority on blast radius, not the person who reported it (Impact Analysis).
- Incident investigation is a legitimate reason to query production values and not a licence to copy them. Pulling a sample of affected rows into a personal notebook creates an uncontrolled copy of exactly the data you are being careful about (PII in Pipelines).
- The incident record itself should name datasets and periods, not example values. A record that quotes rows becomes a long-lived, widely-readable copy of production data with none of its access controls (Data Access Control).
Operating it
- Source total versus serving total for the affected period, which separates "the world changed" from "we lost it" in one query.
- Per-hop row counts for the affected period, which localise a loss to one arrow (Pipeline Metrics).
- The full check suite re-run over the affected period, read as a set rather than individually (Debugging a Data Incident).
- The lineage graph downstream of the suspect dataset, which is the blast radius and therefore the notification list (Lineage Debugging).
- The incident record itself as data: affected datasets, periods, cause, detection source, time to notify, time to repair (Reading a Timeline: Observation Order Is Not Causal Order).
- At ten times the datasets, incidents must be grouped by root cause automatically or the response is spent on triage. Lineage is what makes that grouping possible (Data Lineage).
- At a hundred times the consumers, notification becomes the hard part: there is no channel that reaches everyone reading a table. Surfacing degraded status inside the BI tool, next to the number, is the only mechanism that scales (The Data Quality Dashboard).
- Incident *volume* scales with the number of upstream sources rather than with data size. Every new source is a new schema that can drift and a new producer who does not know you exist (CDC and Schema Drift).
- Diagnosis is dominated by scanned bytes over the affected periods, and by how many times an unsure engineer re-runs a wide query. Bounding every investigative query to the affected partitions is the difference between an incident that costs minutes of compute and one that costs a weekend of it (Scan Cost).
- Repair cost scales with the affected range, not with the size of the bug. A one-character error discovered after a quarter costs a quarter of recompute (Compute Waste).
- The largest cost is not compute at all. It is the decisions taken on the wrong number before anyone was told, and it is the only line that notification latency can reduce.
- Declaring an incident early costs credibility when it turns out to be a business change, and declaring late costs decisions taken on a wrong number. The second is more expensive and the first is more embarrassing, which is why most teams get this backwards.
- Pausing dependent publishes on suspicion protects consumers and makes every downstream dataset stale. That is usually the right trade for tier-1 data and usually the wrong one for exploratory data.
- A thorough affected-range analysis delays the repair. Skipping it produces a fast repair of the wrong extent, which is the version people find out about in a quarterly review.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe candidate-cause list and the order of response — confirm reality, notify, diagnose, scope, repair, add a check — hold for any platform. What varies is how much of the diagnosis is available as a query versus requiring a conversation with a producer.
- ORG-SPECIFICWhether the drop is a business change is answerable in minutes where the data team can query the source system, and in days where it requires a ticket to another department. That single access difference changes the whole shape of incident response.
- SIMPLIFIEDReal incidents frequently have two causes at once — a late batch during a campaign, a schema drift discovered during a backfill — and the single-cause framing here is a teaching simplification. The fingerprint method degrades gracefully: overlapping faults produce a union of failing checks.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns incident command, severity levels, on-call rotation and the post-incident review as a practice. What this lesson adds is the evidence a *data* incident is diagnosed from, which is values rather than logs.