Alert on Symptoms, Not on Causes
"Users cannot check out" beats "CPU is 81%" — with the honest exception of infrastructure conditions that have a specific, immediate response.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Should the pager fire on what users experience, or on the machine conditions that might produce it?
Machine-level signals are easy to collect and easy to threshold, so most alert sets are built from them — and they are only loosely related to whether anything is wrong.
Alert when CPU, memory, disk or queue depth crosses a threshold. These are the things that break, so watching them catches problems early.
High CPU is usually a service doing work. A batch job, a cache warm, a traffic peak and a runaway loop all look identical at the utilisation level, and only one of them is a problem.
- High CPU is usually a service doing work. A batch job, a cache warm, a traffic peak and a runaway loop all look identical at the utilisation level, and only one of them is a problem.
- The converse is worse: most user-visible failures do not raise CPU at all. A bad config, an expired certificate, a dependency returning 500s, a broken query — every one of them can leave every machine metric looking healthy while checkout is completely down.
- Cause-based alerts only detect the causes you thought of. Symptom-based alerts detect failures caused by things nobody anticipated, which is most of them (What Can Fail Between Commit and User).
- Thresholds on utilisation get tuned upward until they stop firing, at which point they no longer detect the case they were added for.
- A wall of green infrastructure dashboards during a total outage is a specific and demoralising experience, and it is what a purely cause-based alert set produces.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A symptom is something a user or a caller experiences: requests failing, requests slow, work not completing, data wrong or stale. A cause is a machine condition: utilisation, saturation, restarts, queue depth, node health.
- The relationship between them is many-to-many and lossy. Many causes produce no symptom because the system absorbed them — that is what redundancy, retries and headroom are for. Many symptoms have no visible cause in your infrastructure, because the cause is in a dependency, in the data, or in the code path itself.
- Alerting on symptoms therefore has a much better ratio: it fires when something is actually wrong, and it fires regardless of which of the many possible causes produced it.
- The cost is that a symptom alert tells you *that* rather than *why*, so investigation starts at scope rather than at cause. Machine metrics do not disappear — they move from the pager to the dashboard, where they are diagnostic rather than notifying (Dashboards an Operator Can Act On).
- The honest exception: an infrastructure condition earns a page when it has a specific, immediate action and waiting genuinely makes it worse. A disk that will be full in an hour, a certificate expiring tomorrow, a replica that has stopped replicating — none is yet a symptom, all have a defined response, and each becomes an outage if ignored.
What the two kinds of alert actually catch
The argument is not that machine metrics are useless. It is that the mapping from machine condition to user harm is weak in both directions, and the pager should sit on the side where the harm is.
Read the last column as the reason the row exists: the case where one kind of alert sees the problem and the other does not.
| Failure | Symptom alert sees it? | CPU / memory alert sees it? | Why |
|---|---|---|---|
| Bad deploy returning 500s | Yes, immediately | No | Erroring fast can use less CPU than succeeding |
| Expired TLS certificate | Yes | No | Nothing on the host changed at all (Certificates as an Operational Object) |
| Dependency timing out | Yes, as latency then errors | Rarely | Waiting is idle, not busy |
| Wrong config after a change | Yes, as errors or wrong behaviour | No | The process is healthy and doing the wrong thing (Configuration Drift) |
| Memory leak approaching the limit | Only once it starts killing pods | Yes, with lead time | This is a real case for a cause alert (OOMKilled: Over the Memory Limit) |
| Disk filling on a stateful node | Only when writes start failing | Yes, with hours of warning | Deadline plus defined action, so it earns a page |
| Batch job at 95% CPU | No — nothing is wrong | Fires, wrongly | Utilisation is the job working |
| Cache returning stale prices | Only with a correctness check | No | Neither kind sees it; this needs a business-level signal |
Rewriting a cause alert as a symptom alert
The practical exercise on an existing alert set is translation: for each cause-based alert, ask what user-visible thing it was meant to protect, and whether that thing has its own alert.
Usually it does not, and adding it lets you demote several cause alerts at once.
page: cpu > 80% for 5m page: memory > 85% page: pod restarts > 3 page: queue depth > 1000 page: db connections > 90 -> five pages during a traffic peak where nothing failed -> zero pages when a config typo broke checkout entirely
page: POST /checkout 5xx > 2%
for 5m, >= 200 req
page: checkout p99 > 3s for 10m
page: order-confirmation job
backlog age > 15m
ticket: db connections > 90%
of budget (action: raise pool)
-> dashboards still show CPU,
memory, restarts, queue depthThe right-hand set fires when purchases are failing, whatever caused it, and stays quiet when the system is merely busy. The machine metrics did not disappear — they moved to the screen the responder opens *after* being paged, which is where they were always most useful.
When an infrastructure alert genuinely earns a page
Being dogmatic here is its own failure. Some machine-level conditions are the right thing to page on, and the test is the same action test as everywhere else: is there something specific to do, and does waiting make it worse?
What these share is a deadline. They are not yet a symptom, they will become one, and there is a defined action in between.
Does the condition have a deadline and a specific action?
when Deadline is hours, action is defined, and the outcome is an outage — disk nearly full on a stateful node, connection budget nearly exhausted, replication stopped.
cost You have woken someone for something that is not yet affecting users, which is justified only if the deadline is real.
when Deadline is days or weeks — certificate expiry, quota headroom, a slowly growing leak that restarts currently absorb.
cost Tickets get deferred; without a date and an owner this is the same as ignoring it (Renewal: Automating the Thing That Expires).
when It is diagnostic context with no independent action — CPU, memory, request rate, restart counts under a threshold.
cost Nobody looks until an incident, which is exactly when it is wanted.
when The condition is normal for this system and has never once required action.
cost You lose the ability to say "it also happened last time"; keep it on the dashboard rather than in the alert set.
How to do it properly
Most important first.
- Start from the user journey. For each critical path — sign in, search, add to cart, check out, receive a webhook — define what failure looks like in a signal, and page on that.
- Express symptom alerts as availability or latency of a specific operation, with a duration and a volume floor, ideally tied to an SLI the team already agreed on (SLIs: Measuring What the User Actually Feels).
- Keep cause-based alerts only where you can name the action and the deadline. Write the action in the alert text: "certificate expires in 5 days; renew via <runbook>" (Renewal: Automating the Thing That Expires).
- Alert on the saturation of things that cannot be recovered quickly — disk, connection limits, quota — because those have a lead time and a specific response (The Connection Budget).
- Include failures your service returns *to* callers and failures it receives *from* dependencies, so a symptom alert exists at each boundary you own (Calling Something You Do Not Control).
- Use synthetic probes for paths whose failure would produce no traffic at all — a broken login page generates fewer requests, not more errors.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing — an alert set that watches only causes fails open: the outage happens and the pager stays silent until a customer complains.
What can go wrong
- Symptom alerts that fire too late because the symptom takes minutes to become statistically visible at low traffic.
- A symptom that never appears in your own metrics because the failure happens before requests reach you — DNS, TLS, CDN, load balancer (How Networks Fail in Production).
- Client-side failures invisible server-side: the API is returning 200 and the page does not render.
- Correctness failures with no error signal at all — the response is 200 and wrong. Prices, permissions and stale caches fail this way, and no latency or error alert will see it.
- Partial-population symptoms averaged away: one tenant, one region or one client version failing while the aggregate stays inside the threshold (Percentiles: Which One, and How Many Users Is That?).
- "Never alert on infrastructure." Wrong in exactly the cases with a deadline and a defined action: disk filling, certificate expiring, replication stopped, quota nearly exhausted.
- "Symptom alerting means we can delete our machine metrics." It means they stop paging. They are still what you read once the symptom alert has fired.
- "If CPU is high, users are affected." Utilisation is not saturation and neither is harm (Saturation: The Reading Utilization Cannot Give You).
- "A 200 response means it worked." Correctness failures are the ones your alerting is least likely to see, and the ones customers notice fastest.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For every critical user journey, you can name the alert that would fire if it broke, and it has fired at least once in a test.
- The last user-visible incident was detected by an alert rather than by a customer, a support ticket or someone noticing a graph.
- The infrastructure alerts that remain each carry a named action in the alert text.
- Moving an alert from paging to dashboard is reversible and should be logged, because a re-added alert usually means the earlier decision was wrong for a reason worth recording.
- Keep the cause-based alerts as non-paging signals when you demote them rather than deleting them outright — they retain diagnostic value on the dashboard even when they should not wake anyone.
- Automate generation of the standard symptom alerts per service from a template — availability and latency of its main operations — so a new service starts with them rather than acquiring them after an incident (Service Templates).
- Automate expiry-based alerting for certificates, tokens and quota, where the deadline is knowable in advance.
- Do not automate the choice of which journeys are critical. That is a product judgement about which failures matter to the business, and it varies per service.
- Symptom alerts detect somewhat later than a perfectly chosen cause alert, because they wait for user impact to be measurable. In exchange they detect causes nobody predicted.
- They require the instrumentation and the journey definition to exist, which is real work that cause-based alerting skips entirely.
- At very low traffic, symptom alerting is statistically weak and you lean more on synthetic probes and absolute counts.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe symptom-over-cause preference follows from the many-to-many relationship between machine conditions and user harm, which holds on any platform. Serverless removes most host metrics entirely and makes the point unavoidable rather than optional.
- SCALE-SPECIFICAt high request rates a small percentage is a clear signal within seconds. At a few requests per minute the same threshold is dominated by single-request noise, so low-traffic services rely on synthetic probes and absolute failure counts instead.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — choosing service level indicators that describe the user journey rather than the machine.