Counters: The Slope Is the Signal
A counter only goes up, which makes the raw value almost useless and its slope almost everything. The two things that go wrong: graphing the total instead of the rate, and mishandling the reset that happens every time the process restarts.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The value is meaningless; the derivative is the metric
A counter answers "how many times has this happened since the process started". Almost nobody wants that number. What they want is "how often is this happening now", which is the slope: take two samples, subtract, divide by the elapsed time. Metrics backends expose this as a rate function, and the window you pass it is the real editorial decision — a 1-minute window is responsive and noisy, a 1-hour window is smooth and slow to react.
This is why counters are the safest default for anything event-shaped. The application does one cheap, lock-free-ish increment and stores no interpretation. The consumer chooses the window at query time, and can choose a different one next year without redeploying anything. Compare that with a requests_per_minute gauge, where the window was frozen into the instrumentation by whoever wrote it first (see Four Metric Types, Four Questions).
The second thing counters give you almost for free is ratios. Two counters over the same window divide into a proportion: rate(errors_total[5m]) / rate(requests_total[5m]) is the error rate, and it is exactly the shape an SLI wants (see SLIs: Measuring What the User Actually Feels). Ratios are more robust than absolutes because they normalize away traffic changes — a doubling of errors during a doubling of traffic is a flat ratio, and usually not an incident.
1# Instrumentation: one increment, no interpretation stored.2http_requests_total{route="/checkout", status="500"}.inc()3 4# Query time — the same series answers all of these:5 6# per-second rate over 5 minutes (responsive)7rate(http_requests_total[5m])8 9# how many in the last hour (absolute count)10increase(http_requests_total[1h])11 12# error ratio — the SLI shape13rate(http_requests_total{status=~"5.."}[5m])14 / rate(http_requests_total[5m])15 16# which route is producing the errors17topk(3, rate(http_requests_total{status=~"5.."}[5m])) by (route)Restarts reset counters, and good rate functions know it
A counter lives in process memory. When the process restarts, it goes back to zero. Naively subtracting consecutive samples across that boundary gives a large negative number, which would render as a nonsensical negative rate.
Well-implemented rate functions detect the decrease and treat it as a reset, counting the new value as the increase since zero. This mostly works and is the reason you rarely see negative spikes on a rate graph. What it cannot recover is the events between the last successful scrape and the restart: if the process crashed 8 seconds after the last scrape, those 8 seconds of increments are simply lost. During a crash loop — which is exactly when you most want accurate error counts — this loss is systematic and biased toward undercounting.
The practical consequence: a counter is an accurate measure of a healthy process and an approximate measure of a crashing one. When you are investigating a crash loop, corroborate counter-derived numbers with something that survives the process, such as gateway-side request counts or log lines shipped as they were written.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Raw `requests_total` on one pod | sawtooth: 0 → 4,100 → 0 → 3,800 → 0 | Each drop to zero is a restart, not a traffic collapse | suspect |
| rate(requests_total[5m]) | 1,190/s, no negative dips | The rate function is handling resets correctly — the shape is plausible | normal |
| Gateway-side request count | 1,310/s | ~10% higher than the pod-derived rate: the gap is the increments lost between last scrape and each crash | smoking gun |
| Process start time gauge | resets every ~90s | Confirms the sawtooth is restarts; correlate with the deploy and OOM signals | smoking gun |
Where counters go wrong in practice
The most common misuse is graphing the total. A steadily climbing line looks like information and carries almost none — it will climb identically whether the error ratio is 0.01% or 4%. Any panel showing a bare counter should be reviewed; the honest version is a rate, an increase over a window, or a ratio.
The second is resetting the counter in application code to "make the graph readable". This destroys the property that makes counters work: the backend can no longer distinguish an intentional reset from a restart, and every window that spans a reset undercounts. If a graph is unreadable, fix the query, not the data.
The third is counting the wrong event. requests_total incremented at the start of request handling counts arrivals; incremented at the end it counts completions, and the two diverge exactly when the system is in trouble and requests are piling up unfinished. Neither is wrong, but the dashboard must say which one it is, and an SLI built on completions will silently exclude every request that timed out (which is precisely the population you care about).
| Smell | Why it misleads | Instead |
|---|---|---|
| A panel showing the raw counter | Climbs monotonically in health and in failure alike | Graph rate() over a stated window, or increase() over the incident window |
| App code resetting the counter periodically | Backend cannot distinguish reset from restart; windows spanning it undercount | Leave the counter monotonic; choose the window in the query |
errors_total with no requests_total | Absolute error counts move with traffic, so the number is not comparable across hours | Store both and graph the ratio — the SLI shape (see SLIs: Measuring What the User Actually Feels) |
| Counter incremented only on success | The failures you most need to count are the ones that never reach the increment | Increment on entry with a status label set at exit, or use a histogram whose count covers all outcomes |
Key points
- A counter's value is nearly useless; its slope over a chosen window is the metric.
- Storing the raw accumulation lets you choose the window — and change your mind about it — at query time.
- Restarts reset counters; rate functions handle the reset, but increments between the last scrape and the crash are permanently lost.
- Ratios of two counters over the same window normalize away traffic changes and are the natural shape for an SLI.
- Increment on entry with an outcome label, or you will systematically fail to count the requests that never finished.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Process → counter: the counter accumulates in memory and is exported at each scrape.
- 2Restart → counter: the process dies, memory is discarded, the counter restarts at zero, and increments since the last scrape are gone.
- 3Scrape → backend: the backend sees a decrease, treats it as a reset, and resumes counting from zero — correctly, but from a value that already omits the lost window.
- 4Backend → dashboard: the derived rate looks smooth and slightly low; nothing on the graph indicates data was lost.
- 5Dashboard → responder: the responder underestimates the error count during exactly the crash loop that needs accurate counting.
- • "The line is going up, so traffic is growing" — a counter line always goes up; only its slope carries information.
- • "No negative spikes, so no restarts" — a competent rate function hides resets by design; check process start time instead.
- • "Errors doubled, this is an incident" — if traffic also doubled, the ratio is flat and nothing has changed for users.
- • "The counter says 4,100 requests, so 4,100 requests happened" — for a process that crashed, the true number is higher by an unknown amount.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Graph `rate(metric[5m])`, and state the window on the panel — the same data looks different at 1m and 1h.
- • Use `increase(metric[window])` to answer "how many during the incident", which is the question postmortems ask.
- • Cross-check pod-derived counters against a gateway or load-balancer count when processes are restarting.
- • Build error ratios as `rate(errors[5m]) / rate(total[5m])` rather than watching absolute error counts.
- • Replace raw-counter panels with rate or increase queries and label the window on the panel.
- • Increment on request entry, set the outcome as a label at exit, so timeouts and crashes are still counted as arrivals.
- • Pair every error counter with a total counter so the ratio is available without a second instrumentation change.
- • When processes restart frequently, corroborate with a counter maintained outside the process (gateway, sidecar, load balancer).
- • Compare `increase(requests_total[1h])` against an independent count (access logs, gateway metrics) — agreement within a percent or two means the counter is trustworthy.
- • Restart a process deliberately in staging and confirm the derived rate stays continuous and non-negative.
- • Check that the error ratio stays flat when you double traffic in a load test without introducing faults.
- • Query-time derivation costs CPU on the metrics backend and makes dashboards slower than reading a pre-computed value.
- • Short rate windows react quickly and produce noisy alerts; long windows are stable and slow to fire. Neither is right for every metric.
- • Counting on entry with an outcome label doubles the label combinations on that metric — real cardinality cost (see [[cardinality]]).
- • Add a dashboard lint rule (or a review convention) rejecting panels that plot a
*_totalseries directly. - • Alert on the error ratio, not the absolute error count, so traffic growth does not silently raise the alert threshold.
- • Track process start-time resets alongside error rates so crash-loop undercounting is visible in the same view.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe sawtooth values and the 10% gap between pod-derived and gateway-derived counts are invented to show the shape of counter loss during a crash loop. The real gap depends on scrape interval and crash frequency.
- ENVIRONMENT-SPECIFICReset handling is a property of the metrics backend's rate implementation. The
rate()/increase()syntax shown is Prometheus-style; other systems express the same derivation differently.