Alert Fatigue: The Page Nobody Reads
Alert fatigue is not a morale problem, it is a detection failure. Every page that turns out to be nothing raises the probability that the next real one is acknowledged late, investigated slowly, or dismissed entirely — and the arithmetic that produces it is measurable.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
How a good alert set decays
Nobody designs a noisy alert set. It accretes, by a mechanism that looks locally reasonable at every step. An incident happens. In the review, someone observes that a particular metric moved beforehand and proposes an alert on it — a well-intentioned suggestion nobody can reasonably oppose. The rule is added. It fires occasionally on that pattern without the incident, but each individual false page is a minor irritation rather than a visible problem, so nothing is done. Repeat monthly for two years.
Two structural facts make this worse than it sounds. First, nothing in the process ever removes a rule: adding an alert is a defensible response to an incident, while removing one requires arguing that a specific failure mode does not warrant detection, which nobody wants to be wrong about. Second, base rates are brutal. A rule that is 95% accurate but fires on a condition that is genuinely a problem 2% of the time produces overwhelmingly false pages — with 100 firings, roughly two are real. Human intuition is bad at this, which is why "the alert is usually right" is a claim worth actually measuring.
The result is a set that is individually justifiable and collectively useless. The end state is predictable: an outage detected by a customer email, while the alert that fired at the right moment sat unacknowledged among four others in a channel nobody had read since Tuesday.
Measuring alert quality
Alert quality is measurable with two borrowed ideas. Precision is the fraction of pages that led to a real action — if 40 pages produced 6 actions, precision is 15%, and the on-call engineer is right to be sceptical of the next one. Recall is the fraction of real incidents that an alert caught before a human or customer noticed. Both matter, and they trade against each other: tightening thresholds raises precision and lowers recall.
Recall is the harder one to measure because it requires knowing about incidents your alerts missed, which by construction you learn about some other way — customer reports, a dashboard glance, another team. Counting those is the point. An organisation that does not track customer-detected incidents has no way of knowing its recall at all, and will systematically over-estimate its coverage.
The most useful single number is actionability: of the pages fired, how many resulted in a human doing something other than acknowledging and closing. It is easy to collect (it is a field on the incident record) and it correlates with everything else. Below roughly 50% it is worth a dedicated cleanup; below 20% the alert set has effectively stopped functioning, whatever its coverage on paper.
| Precision (pages → action) | Recall (incidents caught) | Diagnosis | What to do |
|---|---|---|---|
| High (>70%) | High (>90%) | Healthy alert set | Protect it — review new rules against the four gates |
| Low (<30%) | High | Noisy but covering | Delete and demote cause-based rules; group by root cause |
| High | Low (<60%) | Quiet and blind | Add symptom-based SLI alerts; check for missing async coverage |
| Low | Low | Worst case: noisy AND blind | Rebuild from SLI burn rate; delete the rest and re-add on evidence |
The cleanup, in the order that works
Start by deleting, not by tuning. Take the last 90 days of pages, group by rule, and for each rule count how many led to action. Rules with zero actions in 90 days go — the objection that "it might catch something one day" is exactly the reasoning that produced the current state, and a rule nobody acts on provides no detection anyway. This step alone typically removes a third of the rules.
Then group and suppress. A database failover that fires six alerts is one event; the responder should receive one notification with six symptoms attached, not six pages. Dependency-aware suppression — if the database alert is firing, suppress the six service alerts that depend on it — is more work but removes the worst noise cliffs, which happen precisely when the responder most needs clarity.
Then re-tune what remains, with duration and hysteresis. Most flapping rules are fixed by requiring the condition to hold for a period (for: 10m) rather than firing on a single evaluation, and by separating the firing threshold from the clearing threshold so a metric hovering at the boundary does not oscillate. Only after all three steps is it worth adding anything new — and new rules should be symptom-based, with a runbook, per Alerts Worth Waking Someone For. Finally, make the review recurring: a monthly ten-minute look at pages-versus-actions keeps the set from re-accreting, and is cheaper than the cleanup you would otherwise repeat annually.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Pages per week (before → after) | 41 → 6 | Deleting zero-action rules removed most of the volume | normal |
| Actionability (before → after) | 12% → 78% | A page now usually means something | normal |
| Median time-to-ack (before → after) | 19 min → 3 min | Trust recovered once pages stopped being noise | normal |
| Incidents caught by alert (before → after) | 4/9 → 8/9 | Recall rose while volume fell — they are not opposed | normal |
| Rules with zero actions in 90 days | 23 of 61 deleted | A third of the set was providing no detection at all | smoking gun |
Key points
- Alert fatigue is a detection failure, not a morale complaint: noisy sets measurably raise time-to-acknowledge for real incidents.
- Noisy sets accrete because incidents add rules and nothing removes them, and because low base rates make even accurate rules mostly false.
- Measure precision (pages leading to action) and recall (incidents caught before a human noticed); below ~20% actionability the set has stopped working.
- Clean up in order: delete zero-action rules, group and suppress by root cause, then tune duration and hysteresis. Add last.
- Volume and recall are not opposed — a cleanup usually reduces pages and improves detection at the same time.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Review → rules: each incident adds a cause-based alert; nothing is ever removed.
- 2Base rate → firings: a rule on a condition that is genuinely a problem 2% of the time fires mostly false, however accurate it is.
- 3Firings → behavior: on-call learns that pages are usually nothing; ack time rises from 2 to 19 minutes.
- 4Outage → detection: the correct alert fires and sits unacknowledged among four others in a channel nobody reads.
- 5Customer → support: reports the outage first; the review adds another alert, and the loop closes.
- • Treating fatigue as an on-call attitude problem rather than as a property of the alert set that produced it.
- • Assuming a rule with high accuracy produces mostly true pages — with a low base rate it does not, and the arithmetic is unintuitive.
- • Reading a drop in page volume as reduced coverage; cleanups typically raise recall and lower volume together.
- • Believing tuning thresholds is the main fix, when deletion and grouping remove far more noise per unit of effort.
- • Counting alerts that fired during an incident as detections, without checking whether they fired first.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Group the last 90 days of pages by rule and count how many led to a human action; publish the per-rule table.
- • Record how each incident was detected — alert, dashboard, another team, or customer — so recall becomes measurable.
- • Track median time-to-acknowledge as a trust proxy, and watch it move after cleanups.
- • Count rules with zero firings and rules with zero actions separately: the first is dead weight, the second is active harm.
- • Delete every rule with zero human actions in 90 days; do it as one reviewed change rather than one rule at a time.
- • Group related alerts into a single notification per root cause, and add dependency-aware suppression for the worst cascades.
- • Add duration requirements and separate firing/clearing thresholds to stop flapping.
- • Rebuild the paging core around SLI burn rate so coverage does not depend on enumerating causes ([[burn-rate-alerts]]).
- • Institute a recurring monthly alert review with pages-versus-actions as the standing agenda.
- • Compare pages per week and actionability before and after; both should move in the right direction together.
- • Re-run the incident back-test: confirm recall did not fall after deletions.
- • Watch median time-to-acknowledge over the following month — recovering trust shows up here before anywhere else.
- • Deleting rules occasionally removes the one that would have caught a rare failure — accept it explicitly, since a rule nobody acts on was not detecting anything anyway.
- • Dependency-aware suppression is real engineering work and introduces its own failure mode: a suppression rule that hides a genuinely independent failure.
- • Longer duration requirements delay detection of genuinely fast failures by the duration you chose.
- • Require every new paging rule to name the SLI it protects and carry a runbook, or default it to a ticket (Alerts Worth Waking Someone For).
- • Keep the monthly review permanent; accretion resumes the moment it stops.
- • Add a standing incident-review question: "what fired that should not have?" alongside the usual "what did we miss?".
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll page counts, actionability percentages and before/after figures are teaching numbers showing the shape of a typical cleanup, not measurements from a specific rotation.
- ENVIRONMENT-SPECIFICReasonable page volume depends on rotation size and service count: six pages a week is calm for one team owning one service and alarming for a solo on-call owning twelve.