An Alert Should Demand Action
The rule that decides what is allowed to page a human: if there is no action a person would take right now, it is not an alert.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What earns the right to wake someone up, and what does not?
Alerting is easy to add and socially hard to remove, so alert sets grow monotonically until the pager stops carrying information.
Alert on everything that could indicate a problem. Better a false alarm than a missed outage — we can always tune it later.
The tuning never happens, because tuning means deleting an alert somebody added for a reason, and nobody wants to be the person who deleted the alert before the outage.
- The tuning never happens, because tuning means deleting an alert somebody added for a reason, and nobody wants to be the person who deleted the alert before the outage.
- Every alert that fires without requiring action teaches the responder that alerts do not require action. That lesson is learned reliably and applies to all alerts, including the real ones (Alert Fatigue).
- Alerting on causes rather than effects produces alerts that fire when nothing is wrong — a queue that drains, a node that is replaced, a retry that succeeds — and stays silent when something is wrong through a cause nobody predicted.
- A responder woken for a condition with no defined response does the only thing available: they acknowledge it and go back to sleep, or they restart something and hope.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- An alert is not a measurement, it is an interrupt on a human being. Its cost is not the notification; it is the attention, the context switch, and on a rotation, the sleep.
- That makes the test simple and strict: at the moment this fires, is there something a person should do that cannot wait until business hours? If the honest answer is no, it belongs on a dashboard, in a ticket queue, or nowhere.
- Three properties make an alert actionable. It must be real — the condition is actually happening, not a scrape artefact or a single bad sample. It must be urgent — waiting makes it worse. It must have a response — a person can do something now that improves the situation.
- Severity routing follows from urgency, not from importance. A slowly filling disk is important and not urgent, so it is a ticket. A checkout failure is both, so it pages. Conflating the two is what fills a pager with work that could have waited.
- The threshold question is secondary to the signal question. Choosing 500ms versus 800ms matters far less than whether latency at that endpoint is a thing a user notices at all (Alert on Symptoms, Not on Causes).
The action test
Everything in this lesson reduces to one question asked of every alert, at the moment it would fire: what should a person do right now? Three conditions have to hold together, and an alert that fails any one of them is doing damage rather than nothing.
The test is uncomfortable in practice because most alert sets contain conditions that were genuinely useful once, and removing them feels like removing safety. It is not: an alert nobody acts on is not providing safety, it is providing a habit of ignoring the pager.
| Condition | The alert asks | If it fails | Where it should go instead |
|---|---|---|---|
| Real | Is this actually happening? | Responders learn the alert lies | Add duration and minimum-volume conditions |
| Urgent | Does waiting make it worse? | You have interrupted someone for tomorrow's work | A ticket with an owner and a date |
| Actionable | Is there something to do now? | The responder acknowledges and guesses | A dashboard, or nothing |
| User-relevant | Would a user notice or soon notice? | You page for internal weather | A capacity review (Capacity Management) |
| Owned | Does a specific team respond? | It rings somewhere with no rotation | Fix ownership first (The Ownership Record) |
| Documented | Does the responder know what it means? | Every firing starts from zero | Write the runbook, then enable it (Runbooks) |
Same signal, two alerts
The difference between a good and a bad alert is usually not the metric. It is the conditions wrapped around it and the text a woken human reads at 3am.
Read the two below as what appears on a phone screen. The first requires the responder to reconstruct why they were woken; the second has already done that work.
ALERT: checkout-api
error_rate > 0.05
severity: critical
(no duration, no volume floor)
(no runbook, no dashboard)
-> fires on one failed request at 04:10
when traffic is 12 req/minALERT: Checkout failing for users 5xx on POST /checkout > 2% for 5m, over >= 200 requests affected: ~1 in 50 checkouts dashboard: <link> runbook: <link> recent deploys: checkout-api 03:58 -> fires when purchases are failing
The first alert is technically the same measurement and operationally useless: at low traffic the threshold is met by noise, and the responder has no scope, no suspect and no next step. The second states impact, survives a single bad request, and hands over the two things the responder was going to look up anyway.
How alert sets rot
Alerting rarely fails by being wrong on day one. It fails by accumulating: an incident produces an alert, the alert survives the condition that motivated it, and nobody has authority or appetite to remove it.
These are the specific decay paths. Each has a cheap fix and each is invisible unless someone looks at firing history on purpose.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Incident postmortem adds an alert | Set grows every incident, never shrinks | Adding an alert is the cheapest visible action item | Require the action test in review; prefer fixing the condition to detecting it (Action Items That Change the System) |
| Traffic pattern changes | Alert fires every night at low traffic | Percentage threshold with no volume floor | Add a minimum request count; consider absolute counts at low volume |
| Service is retired or renamed | Alert never fires again | Query returns no data, which satisfies nothing and reports nothing | Alert on absence of data explicitly; audit alerts that have not fired in a long time |
| Deploy causes brief version overlap | Page on every release | Threshold does not tolerate rollout transients (Rolling: Two Versions, One Database) | Widen the duration window past rollout time, or suppress during a known rollout with an expiry |
| Dependency has an outage | Twenty services page at once | Every consumer alerts on the same downstream symptom | Alert at the boundary that owns the dependency; route consumers to a dependency-status signal (Circuit Breaker) |
| Threshold breached routinely and survived | Responders acknowledge without reading | The condition is normal for this system and nobody updated it | Delete or downgrade; a normal condition is not an alert |
How to do it properly
Most important first.
- Apply the action test to every alert, existing and proposed: name the action. If the name is "investigate", ask what would be investigated and whether it can wait.
- Alert on user-visible symptoms, and use cause-based alerts only where the cause has a specific, immediate response (Alert on Symptoms, Not on Causes).
- Require duration and volume conditions. "Error rate above X for N minutes, over at least M requests" removes the two commonest false positives: the momentary spike and the low-traffic percentage.
- Give every alert an owner, a runbook link, and a defined severity that routes it — page, ticket or dashboard — before it is allowed to exist (Runbooks).
- Where the team has an SLO, alert on the rate at which the budget is being consumed rather than on instantaneous breaches: it fires for problems that matter and stays quiet for those that do not (Error Budgets: Unreliability You Are Allowed to Spend, Burn-Rate Alerts: How Fast Is the Budget Going?).
- Review the alert set on a schedule. Every alert that fired and required no action is a candidate for deletion or downgrade, and deleting it is a normal outcome, not an admission.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A missing or ignored alert contains nothing — it converts a small detectable problem into a large one detected by customers.
What can go wrong
- The alert that never fires. An alert nobody has seen work is an untested code path, and the commonest reason is a query that silently returns no data.
- Absence of data read as health. If a service stops emitting entirely, a threshold on its error rate is satisfied forever.
- Alerts that fire during every deploy because they do not tolerate the brief version overlap a rollout creates (Version Coexistence: N and N+1, in Both Directions).
- Flapping: a condition that crosses the threshold repeatedly, producing a page per crossing.
- Alert routing to a group with no on-call, which is functionally the same as no alert but looks configured.
- A dependency's outage firing every downstream service's alerts at once, so the page storm obscures which service is actually at fault (Cascading Failure: When the Response to Failure Causes More Failure).
- "More alerts means we are more likely to catch problems." Past the point where responders trust them, more alerts means less likely — the marginal alert reduces attention paid to all the others.
- "An alert with a runbook is automatically a good alert." A runbook that says "check the dashboard and see if it recovers" is documentation of a page that should not have happened.
- "We alert on CPU because CPU problems cause outages." Sometimes. Most high CPU is a service doing its job, and most outages are not preceded by high CPU (Alert on Symptoms, Not on Causes).
- "Anomaly detection removes the need to decide what matters." It relocates the decision into a model that does not know which anomalies users feel.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For each page in the last month, a responder took an action other than acknowledging it.
- Every alert in the set can be traced to a symptom a user would notice or a condition with a written immediate response.
- Alerts have been deleted recently. A set that only grows is not being reviewed.
- The alerting path itself has been exercised — a deliberate test firing reached the person actually on call.
- Alert definitions belong in version control and roll back like any config, which matters because a badly tuned threshold can page a rotation continuously (A Config Change Is a Production Change).
- Silencing an alert must be time-bounded and must record who silenced it and why. An open-ended silence is an alert that has been deleted without anyone deciding to delete it.
- If you must disable an alert during an incident to stop a page storm, put the expiry in the same action — the commonest way a real signal gets lost is a silence nobody remembered to lift.
- Automate the plumbing: alert definitions as code, reviewed like code, deployed with the service that owns them.
- Automate the mechanical enrichment — owner, runbook, dashboard, recent deploys attached to the notification.
- Automate detection of alert-set decay: which alerts fired, how often, and how many were closed with no action.
- Do not automate the decision to page. Thresholds derived purely from historical distributions page on statistical unusualness, which is not the same as user harm (How to Automate Something).
- A strict action test will eventually miss something that a noisier set would have caught. That trade is deliberate: the noisy set misses things too, by being ignored, and it also costs the team sleep.
- Symptom-based alerting detects problems slightly later than the best cause-based alert would, because it waits for the effect to be visible.
- Good alerting requires the SLO and instrumentation work to have been done first, which is a real prerequisite cost.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe action test holds anywhere a human is interrupted. What changes is what counts as urgent, which follows from what the service promises rather than from the technology.
- ORG-SPECIFICWhich severities exist, where they route, and whether "page" means a phone call or a chat message are conventions set per organisation. The distinction between page-now and ticket-later is real everywhere; the labels for it are not portable (Severity: What It Should Reflect).
- SCALE-SPECIFICBelow a few requests per minute, rate- and percentage-based alerts are dominated by noise — a single failing request is 100% error rate. Small services need absolute-count conditions or synthetic probes instead.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — SLO definition and error budget policy, which decide what "urgent" means before any threshold is chosen.