Quality Alerting
An alert nobody acts on trains people to ignore alerts. Severity from consumer impact, routing to the owner, and the difference between blocking a publish and sending a message.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A check failed. Who should hear about it, how urgently, and should the data have been published at all?
The person who will be woken by this, and the consumer whose decision the failure would have corrupted. An alert that reaches neither of them is a log line with ambitions.
The unit of alerting is one failure of one check on one dataset for one period — not a check, and not a dataset. Alerting per check produces a storm when an upstream model breaks and every descendant fails; grouping by incident rather than by assertion is what keeps the volume survivable (Data Incidents).
Route every failing check to a shared channel. Everything is visible, nothing is missed, and whoever is around can pick it up. This is genuinely the right first step, it costs nothing, and it works for as long as the volume stays low enough that people still read the channel.
One upstream model breaks and forty descendant tests fail. The channel receives forty messages describing one incident, and the useful one — the first — is buried under thirty-nine consequences (Impact Analysis).
- One upstream model breaks and forty descendant tests fail. The channel receives forty messages describing one incident, and the useful one — the first — is buried under thirty-nine consequences (Impact Analysis).
- A distribution check fires every public holiday. After the third time, people close the alert without reading it, and the habit generalises to every alert in that channel (Alert Fatigue: The Page Nobody Reads).
- The alert goes to the data team, who can see that
customer_countryis null for a fifth of new rows and cannot fix it, because the field is populated by another team's service. The alert becomes a ticket, and the ticket becomes a backlog (Who Owns Data Quality). - Everything is at the same severity, so a broken primary key on the revenue model and an unusual row count on a rarely-read table arrive identically. On-call learns the base rate and starts triaging by guessing.
- The failing check did not block the publish, so by the time anyone reads the message the wrong table has been queried by eighty dashboards and exported into three spreadsheets (Stale Dashboards).
What is actually happening
- Alert value is a ratio, not a count: the fraction of alerts that lead to an action. Every alert that leads to no action reduces the attention available for the ones that do, so adding a noisy alert is a *negative* contribution to coverage (Alerts Worth Waking Someone For).
- Severity is not a property of the check. The same uniqueness failure is critical on the model behind a financial report and informational on a table read once a month by one analyst. Severity is derived from consumer impact, which means it depends on lineage, not on the assertion (Data Lineage).
- The block-versus-notify decision is the choice between two failures. Blocking produces staleness, which is recoverable and visible. Not blocking produces wrong data that is live, which is often neither. That asymmetry is why structural checks usually block (Atomic Publish).
- Routing is a lineage question. The team that can fix the cause is upstream of where the check ran, sometimes several hops upstream, and the routing table is the lineage graph with an owner attached to each node (Data Ownership).
- Deduplication and grouping matter more here than in service alerting, because data failures propagate along a DAG deterministically. One root cause producing dozens of downstream failures is the normal case, not the exception (The Transformation DAG).
Severity comes from the consumer, not from the check
The instinct is to assign severity to the assertion: uniqueness failures are critical, volume anomalies are warnings. That is backwards, and it is why so many quality suites page for things nobody would act on and stay quiet about things that matter.
The same failing uniqueness test is an emergency on the model behind a regulatory report and a note on a scratch table that one analyst reads occasionally. Nothing about the check changed; what changed is who is downstream of it. Severity is therefore a lineage property: attach criticality to serving datasets, where consumers actually are, and let it propagate upstream (Impact Analysis).
The table below is the shape of a severity policy that survives contact with an on-call rota. Three levels, each defined by consumer impact, each with a different destination and a different consequence for the publish. Notice that the destination differs by cause as well as by severity — the producer hears about producer-side problems.
Three levels is usually enough. The common failure is not too few levels but too few *distinctions* — one level for everything, so on-call has to reconstruct the consumer impact from the check name at 3 a.m.
| Level | Defined by | Example | Goes to | Publish |
|---|---|---|---|---|
| Critical | A consumer will act on a wrong number, and the wrongness is not visible to them. | Duplicate business keys in the model behind the revenue report; a reconciliation discrepancy on a closed period. | Page the owner of the failing node. If the cause is upstream, page the producing team and inform the data team. | Blocked. Consumers read the previous partition and are told the dataset is held. |
| High | A consumer will notice something is wrong and cannot tell what. | Freshness breach on an operational dashboard; a category that dropped to zero; a completeness gap in an open period. | Ticket to the owning team during working hours, plus a banner on the affected dashboards. | Published, with a visible warning state on the dataset (The Data Quality Dashboard). |
| Informational | Somebody should look before it becomes one of the above. | A row count near the edge of its band; a null rate drifting upward; a check that has not fired in a year. | A channel the owning team reviews on a cadence. Never a page. | Published, no warning state. |
Block the publish, or send a message
This is the decision with the most consequence and the least deliberation. Most teams inherit whatever their tooling does by default and discover the policy during an incident.
The asymmetry to reason from: a blocked publish produces staleness, which is visible, recoverable and correctable by running the pipeline again. A published failure produces wrong data that is live, may already have been exported, and cannot be recalled from the spreadsheets it reached. Between two failures, prefer the one that leaves the previous correct state in place (Atomic Publish).
That argument does not extend to every check. It applies to assertions that are almost never wrong for a legitimate reason — a broken key, a failed contract, a null in a join column. It does not apply to assertions that depend on judgement about the business, which are wrong often enough that blocking on them makes the platform unreliable in the ordinary sense.
If this check fails, is a stale table better for consumers than a published one?
when The failure makes the table unusable rather than merely suspicious: duplicate business keys, nulls in a join key, a contract violation, a referential break.
cost Staleness until a human unblocks it, and a human who must be available. Requires the publish to be atomic, or blocking is not actually possible (Contract Enforcement).
when Structural checks pass but a reconciliation on a closed period disagrees with the source, on a dataset whose numbers get reported externally.
cost Reconciliation runs after the period closes, so blocking here delays a publish that consumers were already waiting for. Worth it only where a wrong reported figure is costly to retract (Reconciliation).
when The check needs human judgement — a distribution anomaly, a freshness breach on a source that is sometimes legitimately quiet.
cost Consumers must actually see the warning, which means it has to appear where they read the data, not only in a channel they do not open (The Data Quality Dashboard).
when The dataset is exploratory, has few consumers, or the check is new and its false-positive rate is unknown.
cost Bad data is live between publish and action. Acceptable when the consumer is one analyst who will notice, and not when it is an automated model (Feature Pipelines).
when The signal is worth having as a series but is not actionable on its own — drift indicators, coverage counts, checks in their tuning period.
cost Nobody looks unless something else brings them there. This is the correct home for most new checks for their first few weeks (Pipeline Metrics).
Routing: the alert goes to whoever can fix it
The most common structural mistake in data alerting is that every failure, wherever it originated, is delivered to the data team. That team can see the problem, describe it precisely, and in a large fraction of cases do nothing about it, because the cause is a field populated by a service they do not own.
Routing correctly requires two things that are boring to build and decisive in practice: an owner attached to every node in the lineage, and a rule that maps the failing check to the nearest upstream owner rather than to whoever runs the pipeline. With those, a null-rate failure on customer_country reaches the team whose service stopped populating it (Data Ownership).
The diagram is the routing rule as a picture. Two checks fail on the same model; one is a producer-side problem and one is a transformation bug, and they must not go to the same place. Getting this right is what converts a quality programme from a reporting function into something that reduces failures (Who Owns Data Quality).
How to build it
Most important first.
- Derive severity from the consumer, not from the check. Attach a criticality to each serving dataset, propagate it upstream along the lineage, and let the check inherit it (Impact Analysis).
- Decide block-versus-notify per check, and write the rule down: structural assertions that make the table unusable block; shape assertions that need human judgement notify (Data Tests).
- Route to the team that can fix the cause. If the check found a producer-side problem, the producer gets the alert and the data team is copied — not the other way round (Who Owns Data Quality).
- Group by incident. When a model fails, suppress its descendants' alerts and report one incident with a blast radius rather than one message per assertion (Lineage Debugging).
- Make every alert carry the query that reproduces it and the keys of a sample of offending rows. An alert that says "uniqueness failed" without saying on what has delegated the first ten minutes of the investigation to the reader (Structured Logging: Fields a Program Can Read).
- Review fired-versus-actionable per check on a schedule, and delete or widen the checks that lose. A quality suite needs pruning as much as it needs additions (Alert Fatigue: The Page Nobody Reads).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A blocking check guarantees consumers never read a table that violates it. That is the only strong guarantee in this lesson and it is bought with availability (Contract Enforcement).
- A notifying check guarantees only that a message was produced. Whether it was read, understood, or acted on is outside the system, and assuming otherwise is the central error here.
- Routing guarantees nothing about ownership. Sending an alert to a team does not make them responsible for it, and a routing table built without that agreement produces ignored alerts with correct addresses (Data Ownership).
- Nothing here guarantees detection. Alerting is the delivery layer for the checks you wrote; the failures nobody asserted produce no alert of any severity (Data Quality).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check on the alerting itself: does a deliberately broken staging dataset produce exactly one alert, at the right severity, to the right team, within the expected window? Test the routing the way you would test a failover.
- Track alerts fired, alerts acknowledged, and alerts that led to a change, per check. The third number is the only one that matters and the only one usually not recorded (Reading a Timeline: Observation Order Is Not Causal Order).
- It misses the alert that was never configured — a dataset with no criticality attached inherits nothing and routes nowhere, and the failure is a check that fires into a void.
- Blocking checks convert a correctness problem into a freshness problem. The dataset becomes stale until somebody unblocks it, which is why blocking requires a human who is actually available (The Freshness SLO).
- Alert latency matters differently by severity. A critical failure on a table feeding an operational decision needs to be known within the decision's own timescale; a monthly model can be told tomorrow.
- The slowest and strongest checks — reconciliation on a closed period — produce alerts that arrive after the reported number has circulated. That is unavoidable, and it means the alert's purpose is correction and communication rather than prevention (Reconciliation).
- Criticality changes when consumers change. A model that gains an executive dashboard has become critical without anybody editing the pipeline, and the severity should follow automatically from the lineage rather than from memory (Impact Analysis).
- Routing decays as teams reorganise. An owner field that points at a team that no longer exists is worse than an empty one, because it looks configured (Data Ownership).
- Thresholds drift into uselessness after each false alarm. Record why a threshold was changed, so that a suite that has been quietly negotiated to silence is visible as such (Distribution Tests).
- Recovery from a blocking failure is: fix the cause, re-run, publish, and confirm the previous partition was never replaced. The dataset was stale, not wrong, which is the recoverable failure (Atomic Publish).
- Recovery from a non-blocking failure includes a communication step that engineers routinely skip: consumers who read the bad data need to be told, and the lineage tells you who they were (Data Incidents).
- Recovery from alert fatigue is not a technical change. It is deleting checks, and it works only if the deletion is deliberate and recorded rather than achieved by muting (Alert Fatigue: The Page Nobody Reads).
What can go wrong
- One root cause producing dozens of alerts, so the signal is buried in its own consequences.
- A single severity level, which forces on-call to triage by guessing.
- Alerts routed to a team with no ability to fix the cause, converting detection into a ticket queue.
- A blocking check with no available human to unblock it, turning a minor quality failure into a prolonged outage (Fail Open vs Fail Closed).
- Muting rather than fixing, which removes coverage invisibly — the failure of the mitigation itself.
- An alert that fires correctly into a channel nobody has read since the team reorganised.
- "More alerts means better coverage." Alerts that lead to no action reduce coverage by consuming the attention the useful ones need.
- "The alert fired, so the process worked." The process worked when somebody acted. Firing is the cheap half.
- "We should block on everything." Blocking on checks that are legitimately wrong once a fortnight produces a platform people route around, which is worse than not blocking (Data Platform Anti-Patterns).
- "Nobody complained, so the alerting is fine." Nobody complaining is also what silence looks like when every alert is being closed unread (Alert Fatigue: The Page Nobody Reads).
Operating it
- Alert volume per check per week, with the actionable fraction alongside it. A check below a useful ratio is costing more attention than it returns (Alerts Worth Waking Someone For).
- Time from check failure to acknowledgement, per severity, which is how you find out whether the routing works at all (Reading a Timeline: Observation Order Is Not Causal Order).
- Count of datasets with no criticality and no owner attached — the coverage gap that produces alerts with nowhere to go (The Data Catalog).
- Publishes blocked per week, which is the direct measure of what blocking checks are costing in freshness (Error Budgets: Unreliability You Are Allowed to Spend).
- At 10x datasets, per-dataset alert configuration is not maintained. Severity and routing must be derived from declared ownership and lineage or most datasets will have neither (Metadata: Technical, Operational and Business).
- At 100x checks, alert grouping stops being a nicety and becomes the only thing that keeps the channel readable — one incident with a blast radius instead of a hundred assertions (Impact Analysis).
- Consumer growth raises severity without raising check count. The same model becomes critical because eighty dashboards now read it, and nothing in the pipeline changed (Data Discovery).
- The dominant cost is human attention, and it is spent whether or not the alert was useful. This is the one cost in the module that never appears on an infrastructure bill and always appears in retention (Alert Fatigue: The Page Nobody Reads).
- Blocking costs availability. Every blocked publish is consumers waiting, and at some threshold of frequency they will build a path around the blocked table (Data Platform Anti-Patterns).
- Grouping and deduplication cost engineering time to build and repay it during the first incident where one model breaks forty descendants (The Transformation DAG).
- Blocking trades availability for correctness, and the trade is right far more often than teams are comfortable with — but it requires someone to be available, which is an organisational commitment rather than a configuration.
- Severity derived from lineage is correct and opaque: nobody can explain why a check is critical without reading the graph, which makes disputes harder to resolve than a hand-written label would.
- Aggressive pruning of noisy checks improves the signal ratio and removes coverage. Both effects are real, and the pruning is still usually right — a check nobody reads provides no coverage anyway (Alert Fatigue: The Page Nobody Reads).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe relationship between alert volume, actionable fraction and attention holds for any monitoring system. What differs from service alerting is that data failures propagate deterministically along a DAG, so grouping by root cause is both more necessary and more tractable here.
- ORG-SPECIFICWho receives a quality alert and whether they can act on it is decided by ownership boundaries, not by tooling. In a platform team model the alert routes internally and stays a data-team problem; where producers own their outputs it routes to the producing service team, and the same check produces a completely different workflow.
- TOOL-SPECIFICWhether a failing test can actually block the publish depends on the orchestration: in tools where tests are DAG nodes whose failure halts descendants, blocking is available; where tests run after the model materialises, the only options are notify and roll back. Confirm which you have before promising prevention.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the on-call machinery this plugs into: rotations, escalation policies, paging, and the review that decides whether an alert earned its page. This lesson decides what to send and to whom; that domain decides how it is delivered and who is awake.