Log Levels Are a Convention, Not a Standard
Nothing in any specification says what warn means. What it means is whatever your team decided, written down or not — and when it was never written down, everything becomes info, the error rate becomes unmeasurable, and the level field stops carrying information.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Four levels, and the only question that decides them
The useful discriminator is not severity in the abstract — it is who needs to act, and when. error means a human must look at this, ideally soon. warn means something recoverable happened that is worth knowing if a pattern forms, but nobody is woken up. info records the normal operation of the system so a request can be reconstructed afterwards. debug is detail that is too voluminous to keep on by default.
Applied consistently, this makes level queryable. rate(level="error") becomes a real health signal, and an alert on it means something. Applied inconsistently, the field degenerates: half the errors are at info because they felt routine, half the warn lines are actually normal, and the only honest query is one that ignores level entirely and filters on event instead (see Structured Logging: Fields a Program Can Read).
The specific trap worth naming is the expected failure. A validation rejection, a 404, a retry that succeeded on the second attempt — these are the system working correctly. Logging them at error is the fastest way to make the error level meaningless, because their volume dwarfs real failures and trains everyone to ignore the level.
| Level | Decision rule | Example | Common misuse |
|---|---|---|---|
error | A human must investigate; something is broken that we intended to work | Payment provider unreachable after all retries | Expected validation failures, 404s, user typos |
warn | Recoverable, but a pattern here would matter | Retry succeeded on attempt 2; falling back to a replica | Anything nobody would ever act on, which is just info |
info | Normal operation worth reconstructing later | Request completed, job finished, config loaded at startup | Per-iteration loop detail, which is debug |
debug | Detail too voluminous to keep on by default | Full request bodies, per-item processing steps | Left enabled in production "temporarily" |
Everything-is-info, and the error rate that lies
The most common end state is that almost everything is info. It happens gradually and for a reasonable-sounding reason each time: an author is unsure, info is the safe default, nobody objects in review. Once the ratio is far enough gone, level conveys nothing and every query has to filter on event names instead.
The mirror image is just as damaging: routine failures logged at error inflate the error line rate by an order of magnitude, so an alert on that rate either fires constantly or is set so high it never fires. This is the log-shaped version of the problem Alert Fatigue: The Page Nobody Reads describes — a signal that cried wolf until nobody looks.
The corrective is cheap. Write the decision rules down, put them in the shared logging helper's documentation, and check the level distribution periodically. A healthy service usually shows info dominating, warn a small fraction, and error genuinely rare — and if the shape shifts sharply after a deploy, that is a signal worth reading alongside "What Changed?" — Deploy Markers and the Invisible Deploys.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Healthy service: info / warn / error | 94% / 5% / 1% | Level carries information; an alert on error rate is meaningful | normal |
| Degenerate service: info / warn / error | 99.8% / 0.1% / 0.1% | Almost everything is info — level is not a usable filter here | suspect |
| Inflated-error service: info / warn / error | 61% / 3% / 36% | Routine failures are being logged as errors; no threshold on this rate can work | smoking gun |
| Error rate before/after deploy | 1.0% → 12% | Either a real regression or a logging-level change — check the event breakdown before concluding | suspect |
| Debug lines in production | 4.1% of volume | Debug was enabled and not turned off; a cost and privacy issue both | suspect |
Changing the level without a deploy
During an incident you frequently want more detail from one component, right now. The choice between "ship a deploy to enable debug logging" and "give up and guess" is a bad one, especially since the deploy itself restarts processes and may clear the very state you were investigating.
Runtime-adjustable levels solve this: a control that raises verbosity for a specific logger, ideally scoped to one service, one component, or a sampled fraction of requests. Scoping matters because global debug on a busy service can multiply log volume by ten or more and create a second incident in the log pipeline (see The Log Bill and What It Is Buying).
Two safeguards make this safe to hand to on-call. A time-boxed default, so elevated verbosity expires automatically rather than being forgotten. And an audit record of who changed what, since debug output is exactly where sensitive values are most likely to appear (see What You Just Wrote Into a Log Half the Company Can Read).
1# Bad: global, permanent, no audit2set_log_level("debug")3 4# Better: scoped to one component, expires on its own5set_log_level(6 logger = "checkout.payment", # not the whole service7 level = "debug",8 duration = "15m", # auto-reverts; cannot be forgotten9 sample = 0.05, # 5% of requests, not all of them10 reason = "INC-4417 provider timeouts",11 actor = "oncall@example.com", # audited: debug output may contain sensitive values12)13 14# Volume check before enabling globally:15# info-level volume ~12 GB/day16# estimated at debug ~140 GB/day17# ingest quota 25 GB/day18# -> global debug would drop data for 4 hours. Scope it.Key points
- No specification defines the levels; the only thing that gives them meaning is a written team convention.
- Decide the level by who must act and when, not by how bad the event feels in the abstract.
- Logging expected failures at
errordestroys the error level's usefulness faster than anything else. - The level distribution is itself a signal — a sharp change after a deploy is worth investigating.
- Runtime-adjustable levels should be scoped, sampled, time-boxed and audited, or they create a second incident.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Author → log call: unsure which level applies, picks
infoas the safe default. - 2Log call → distribution: over months the
infoshare climbs toward 100% anderrorbecomes rare and arbitrary. - 3Distribution → query: filtering on
levelno longer separates real problems from normal operation. - 4Query → alerting: an alert on error-line rate is either constantly firing or set so high it never does.
- 5Alerting → responder: the responder stops trusting the level field and filters on
eventnames instead, which only works for events that have stable names.
- • "Error count doubled, we have a regression" — check whether a deploy changed what gets logged at
errorbefore concluding anything. - • "No errors in the logs, so nothing is wrong" — the failure may be sitting at
infobecause the author was unsure. - • "We should log more at error so problems are visible" — that dilutes the level and makes every real error less visible.
- • "Debug is off in production" — verify it; a temporary enablement left on is common and shows up as both cost and privacy exposure.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Compute the share of lines at each level per service and compare against the team's stated intent.
- • Count distinct `event` values at `error` — a long tail of routine events means the level has been diluted.
- • Check for debug-level lines in production, which usually means a temporary change was never reverted.
- • Estimate log volume at debug before enabling it, and compare against the ingest quota.
- • Write the level convention down with a decision rule per level and put it where the logging helper is documented.
- • Move expected failures (validation, 404s, successful retries) down to `info` or `warn` and keep `error` for things needing investigation.
- • Add scoped, sampled, time-boxed runtime level control so incidents do not require a deploy.
- • Review the level distribution periodically and treat sharp changes as a signal, not as noise.
- • Confirm the level distribution moves toward the intended shape and that error-level `event` names are all genuinely actionable.
- • Verify an alert on error rate now correlates with real incidents rather than with traffic volume.
- • Test that elevated verbosity auto-reverts after its window and that the change was audited.
- • A strict convention slows down writing log calls slightly and needs enforcement to survive team growth.
- • Runtime level control is another control-plane surface with its own auth, audit and failure modes.
- • Moving expected failures off `error` means a genuinely novel failure may be under-noticed if it lands at `warn`.
- • Alert if debug-level volume appears in production above a small threshold.
- • Include level choice in code review for new log calls, especially anything at
error. - • Track the error-level
eventcardinality; growth means routine events are creeping back in.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVELevel distributions, the volume estimates and the ingest quota are invented to give the ratios a shape. Healthy distributions vary enormously by service type — a batch worker and an API server look nothing alike.
- ENVIRONMENT-SPECIFICAvailable levels, their names and whether runtime adjustment exists are properties of the logging library and platform. Some stacks add
trace,fatalor numeric levels with different conventions again.
Misconceptions
warn, which is why the convention has to be written down to exist at all.error, no threshold works and the real failures are less visible than before.