Alert Fatigue
Noise trains responders to ignore the pager, and the cost is paid on the one night the alert was real.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does a noisy alert set actually cost, given that each individual false page is only a minor annoyance?
The damage from a noisy alert is not the interruption. It is the learned response — acknowledge without reading — which transfers to every alert including the ones that matter.
The pager is noisy but people cope. Responders learn which alerts to take seriously, so the extra ones are harmless.
Responders do learn which alerts to take seriously, and they learn it from firing history rather than from severity labels. An alert that has been wrong twenty times is treated as wrong the twenty-first, when it is right.
- Responders do learn which alerts to take seriously, and they learn it from firing history rather than from severity labels. An alert that has been wrong twenty times is treated as wrong the twenty-first, when it is right.
- The learning is not per-alert in practice. A rotation that is woken frequently for nothing develops a general expectation that pages are not real, and the delay applies to all of them.
- The cost is invisible until the miss. Nothing in the metrics distinguishes "alert acknowledged in 30 seconds and correctly ignored" from "alert acknowledged in 30 seconds and wrongly ignored" — until the incident review.
- Sustained noise drives people off the rotation. That is a staffing and retention cost, and it removes exactly the experienced responders who could tell a real page from a false one (Rotations People Can Sustain).
- Under a page storm, the real signal is present and unfindable: twenty alerts arrive within a minute and the one naming the actual cause is somewhere in the middle.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The mechanism is ordinary and human: a signal that is usually wrong stops being treated as information. This is not a discipline failure and cannot be fixed by asking people to be more careful. It is the correct response to an unreliable signal.
- Response time degrades before detection fails outright. The first symptom of fatigue is not a missed page, it is a slower acknowledgement and a slower first action — visible as rising time-to-mitigate long before anything is missed.
- Noise compounds through the rotation. Sleep loss reduces judgement, which lengthens the next incident, which produces more alerts. A rotation in this state is degrading its own reliability (Rotations People Can Sustain).
- Storms are a distinct problem from steady noise: a single upstream failure fires every downstream alert, and the responder has to triage the alert list before triaging the incident (Cascading Failure: When the Response to Failure Causes More Failure).
- The organisational cause is nearly always the asymmetry in how alerts are added and removed. Adding one takes minutes and looks responsible; removing one requires someone to accept the risk publicly.
The path from noise to a missed incident
The chain below is not hypothetical; it is the ordinary lifecycle of an unmanaged alert set. Every step is individually reasonable, and the endpoint is a system that has monitoring and no detection.
Note where it can be broken. The cheapest intervention is at the second step — before the response has become automatic — and the most expensive is at the last, which is a postmortem.
What to do with a noisy alert
Deleting is one option among several, and often not the best one. The information in a noisy alert is usually real; what is wrong is that it interrupts a person.
Pick by what the alert is actually telling you, not by how annoying it has been.
What is the right disposition?
when The condition is normal for this system and carries no diagnostic value.
cost You lose the historical signal; if unsure, demote instead of deleting.
when It represents real work that does not need doing tonight.
cost Requires a queue with an owner, or it is deletion with extra steps.
when It is diagnostic context, valuable during an incident and not before one.
cost Nobody sees it proactively.
when It fires because something is genuinely wrong and has been tolerated — a leak absorbed by restarts, a job that always retries.
cost Real engineering effort, and it is the only option that removes the underlying problem.
when The signal is right and the trigger is too twitchy.
cost Detection is slightly later; usually the correct trade.
when It is one of many that fire together from a shared cause.
cost A concurrent unrelated incident can hide inside the group.
The measurements that make it discussable
Fatigue arguments go in circles when they are about how the pager feels. They resolve quickly when there is a table of which alerts fired and what happened next.
These four are enough. They are also the only place in this module where you should be counting anything, and the counts are of your own alert set, not of incidents in general.
| Measure | What it tells you | What it does not tell you |
|---|---|---|
| Firings per alert | Which single alert dominates the noise | Whether those firings mattered |
| Actions per firing | Whether an alert has any operational value | Whether a rare firing would have mattered |
| Pages per shift, in and out of hours | The human load the rotation is actually carrying | How hard each page was to handle (Rotations People Can Sustain) |
| Time from page to first action | Whether trust in the pager is degrading | Whether the response was correct |
| Alerts fired per incident | How much storm triage the responder had to do first | Which of them was the useful one |
How to do it properly
Most important first.
- Measure the alert set, not the opinions about it: firing count per alert, and how many firings resulted in an action. An alert that fired repeatedly with no action is the finding.
- Give someone explicit authority to delete alerts, and make deletion a normal, logged, reversible act rather than an argument.
- Fix the top offender each week rather than attempting a whole-set cleanup. One alert usually dominates the count.
- Group and deduplicate at the notification layer so a dependency failure produces one page with a list, not twenty pages (Telling People What Is Happening).
- Route by urgency honestly. Most of a noisy set is work that could wait until morning; moving it to a ticket queue keeps the information and removes the interruption.
- Track the noise level as an operational metric that the team owns, alongside deploy frequency and time-to-mitigate — it is a property of the system, not a complaint (Toil).
- Review alerts fired during each incident in the postmortem: which helped, which distracted, which were missing (Postmortems).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains it — fatigue degrades detection for every service the rotation covers, simultaneously and invisibly.
What can go wrong
- Cleanup that goes too far and removes the alert that would have caught the next class of failure. Demote to ticket or dashboard rather than delete when unsure.
- Silences with no expiry, which turn a noisy alert into a silently absent one.
- Deduplication that groups too aggressively and hides a second, unrelated incident inside the first.
- Threshold relaxation as the fix for every noisy alert, until the threshold is above any level the condition ever reaches.
- Treating fatigue as an individual failing — "the responder should have looked more carefully" — which leaves the system unchanged and adds a person to blame (Root Cause vs Contributing Factors).
- "Fatigue is a discipline problem." It is a signal-quality problem. Ignoring an unreliable signal is rational behaviour and will not be trained away.
- "We will clean up the alerts after this quarter." The cleanup that never happens is the mechanism by which every noisy alert set was created.
- "Severity labels solve it." Responders calibrate on firing history, not on labels. A critical alert that is usually wrong is treated as usually wrong.
- "Nobody has missed anything, so the noise is affordable." Detection degrades continuously and fails discretely; the absence of a miss so far is not evidence of margin.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The proportion of pages that led to a real action is known, and is high.
- Pages per person per shift is tracked, and nights are quiet often enough that a page is genuinely unexpected.
- Alerts were removed or downgraded in the last quarter, and the decisions are recorded.
- Time from page to first action is stable or improving, rather than drifting upward.
- Every deletion or downgrade is a config change in version control and can be reverted in a minute if the next incident shows it was load-bearing (The Audit Trail).
- Prefer a staged retreat — page becomes ticket, ticket becomes dashboard — so a reversal is one step rather than a rewrite.
- Automate the accounting: firings per alert, actions per firing, pages per shift. Nobody will assemble this by hand, and without it the discussion is anecdotal.
- Automate grouping, deduplication and dependency-aware suppression so one root failure produces one notification.
- Do not automate deletion. An alert with zero firings might be dead, or might be guarding a rare and severe condition (The Automation Trap).
- A quieter set has a higher chance of missing an unusual failure. It has a much lower chance of missing an ordinary one, because someone is actually reading the pages.
- Grouping reduces noise and adds a small delay, and can mask a second incident that begins during the first.
- Doing this properly costs sustained attention on something that never appears as a feature, and it competes directly with delivery work.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe response to an unreliable signal is human and universal. What varies is how quickly it sets in, which depends on how often the rotation is interrupted outside working hours.
- ORG-SPECIFICWhether alerts can be deleted at all is an organisational question. In environments where an alert was added by an audit or compliance requirement, the available move is routing it to a reviewed queue rather than a pager, which requires agreement rather than a config change.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — treating detection quality as a tested property of the system rather than as a property of how attentive the responder was.