ObservabilityGENERALFRAMEWORK-SPECIFICRUNTIME-SPECIFIC

The Metrics a Backend Must Emit

Request rate, error rate, latency, in-flight requests, pool usage, queue depth, cache hit rate and dependency latency — eight numbers that make a service legible.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

Which numbers does a backend have to publish for anyone to know whether it is healthy?

The requirement

The service is "up" — the process is running and the health check is green — and customers say it is slow. We need numbers that describe behaviour, not existence.

The obvious build

CPU and memory are already collected by the platform, and we have logs for everything else. If something breaks, we will search the logs.

Why it breaks

CPU and memory describe the *process*, not the work. A service can sit at 20% CPU and be completely stalled waiting on a saturated connection pool (Connection Pool Saturation: Waiting in Front of an Idle Database).

How it breaks in production

CPU and memory describe the *process*, not the work. A service can sit at 20% CPU and be completely stalled waiting on a saturated connection pool (Connection Pool Saturation: Waiting in Front of an Idle Database).
Logs are per-event and expensive to aggregate. Asking "what is the error rate right now" by counting log lines is slow, costly, and cannot be the basis of a real-time alert.
Without in-flight and queue-depth numbers, saturation is invisible until it becomes latency — and by then the queue is deep and recovery takes far longer than the incident (Queueing: Why Systems Get Slow Before They Get Broken).
Averages hide the problem: mean latency stays flat while the slowest 1% of requests — often an entire customer segment — goes to seconds (The Average Was Fine and Users Were Not).
Nothing distinguishes "our code got slower" from "the payment provider got slower", so every incident starts with an argument.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

A metric is a cheap numeric aggregate emitted continuously, in contrast to a log, which is an expensive record of one event. That difference decides what each is for.
Three instrument types cover almost everything: counters monotonically increase and are read as rates; gauges go up and down and are read as current values; histograms bucket observations so percentiles can be computed (Four Metric Types, Four Questions).
The eight numbers below are not a taxonomy someone invented; each answers a distinct operational question that the others cannot.
Request rate, error rate and latency distribution are the request-facing three — the RED shape. Together they describe what callers experience (RED: Rate, Errors, Duration).
In-flight requests, DB pool usage and queue depth are saturation signals: they show a resource filling up *before* it manifests as latency (Saturation: The Reading Utilization Cannot Give You).
Cache hit rate and dependency latency/error rate are attribution signals: they say where the time and the failures are coming from.
Labels (dimensions) make a metric sliceable, and each distinct label combination is a separate stored time series. High-cardinality labels multiply series count, which is the dominant cost and stability risk in any metrics system (Cardinality: The Label That Took Down Monitoring).
Percentiles do not average. The p99 of two instances is not the mean of their p99s — this is why latency must be recorded as a histogram and aggregated as buckets, not pre-computed per instance (Percentiles: Which One, and How Many Users Is That?).

Eight numbers and the question each one answers

The test for whether a metric belongs on this list is whether removing it makes a specific class of incident undiagnosable. Each row below fails a different investigation when it is missing, which is why the list is short and why it is not shorter.

The first three describe what callers experience. The middle three describe resources filling up, and they are the ones that give warning. The last two attribute the problem to somewhere other than your own code.

Signal	Instrument	Answers	Missing it means
Request rate	counter, by route + method	Is traffic normal? Did a caller change behaviour?	A traffic drop looks like nothing at all
Error rate	counter, by route + category	Are we failing, and how?	Outages found via customer tickets (An Error Taxonomy That Maps Cause to Response)
Latency distribution	histogram, by route	What do slow users experience?	Averages hide the tail (Percentiles: Which One, and How Many Users Is That?)
In-flight requests	gauge	Are we saturated right now?	No warning before latency climbs (Little's Law as Working Intuition)
DB pool in-use + wait time	gauge + histogram	Are requests queueing for a connection?	A stalled service at 20% CPU and no explanation (Connection Pools)
Queue depth + oldest message age	gauge	Are consumers keeping up?	A backlog discovered when it is hours deep (Queue Backlog)
Cache hit rate	counter pair	Is the cache doing anything?	A silently useless cache adding a consistency problem (When Not to Cache)
Dependency latency + errors	histogram + counter, by dependency	Is it us or them?	Every incident starts as an argument (Calling Something You Do Not Control)

Labels are the cost, and the trap

Every distinct combination of label values is a separate time series stored for the retention period. Route pattern times method times status class is a bounded, small product. Add a user id and the product becomes unbounded — one series per user, forever, including users who made one request in 2023.

The failure is not gradual. Metrics backends degrade sharply past their capacity, and the degradation happens during the traffic spike that added the cardinality, which is also the incident you needed the metrics for.

Bounded labels versus a cardinality bomb

1// GOOD: bounded label space.
2// routes (~40) x methods (~4) x status classes (5) = a few hundred series.
3httpRequests.inc({
4  route: '/orders/:id',        // the PATTERN, from the router
5  method: req.method,
6  status_class: `${Math.floor(res.statusCode / 100)}xx`,
7})
8 
9httpDuration.observe(
10  { route: '/orders/:id', method: req.method },
11  durationSeconds,
12)
13 
14// BAD: unbounded. Each of these adds one series per distinct value, forever.
15httpRequests.inc({
16  route: req.originalUrl,      // /orders/8fa2... -> one series per order
17  user_id: req.auth.userId,    // one series per user
18  error: err.message,          // one series per unique error string
19})
20 
21// The identifying detail belongs in a log line, where high cardinality is fine
22// and you pay per event rather than per series.  [[structured-logging]]

The dividing line is whether the set of possible values is bounded by your code (routes, methods, statuses, dependency names) or by your data (ids, paths, messages). Bounded goes in labels; unbounded goes in logs and traces.

Saturation is the signal that arrives early

Latency is a lagging indicator. By the time p99 has doubled, the queue is already deep and the system needs time to drain even after the cause is removed. Saturation gauges move first, and they move while everything still looks fine to callers.

The relationship is not mysterious: in-flight requests equal arrival rate times average latency. When a resource saturates, service time rises, in-flight count rises with it, and the queue grows — the depth on finding and modelling that behaviour lives in Observability & Performance (Little's Law as Working Intuition, Queueing: Why Systems Get Slow Before They Get Broken).

What each saturation signal catches before latency does

Trigger	Symptom	Cause	Response
A query loses its index after a migration	Pool wait time rises; in-flight climbs; p99 still normal for a few minutes	Each request holds a connection longer, so the fixed pool serves fewer concurrent requests	Alert on pool wait time > 0 sustained, not on pool utilisation (Connection Pools)
A consumer deploy fails and pods crash-loop	Queue depth flat, oldest-message age climbing steadily	Depth can look stable while nothing is being consumed if producers also slowed	Alert on message age, not depth — age is the signal that cannot lie (Queue Backlog)
A JSON payload grows tenfold after a client change	Event-loop lag rises; in-flight climbs; CPU rises last	Serialization is CPU work on the loop thread, blocking every other request	Watch loop lag as a first-class gauge on Node (Blocking the Event Loop)
A dependency slows from fast to merely slow	Dependency latency histogram shifts; your in-flight count doubles	Your concurrency is bounded by their response time, not by your CPU	Bound concurrency per dependency and shed rather than queue (Bulkheads)
Cache node evicted or restarted	Hit rate falls to zero; DB load and pool wait rise together	Every request now reaches the origin; a stampede may follow	Alert on hit-rate drop as a leading indicator of DB saturation (Cache Stampede)

How to build it

Most important first.

Start with RED per route: request count, error count and a latency histogram, labelled by route pattern, method and status class. This is the smallest set that makes a service legible.
Add the saturation gauges next: in-flight requests, pool in-use versus size, pool wait time, queue depth and queue age. These are what turn a post-mortem into a warning (Backpressure).
Instrument every outbound dependency identically — call count, error count, latency histogram, labelled by dependency and operation. Without this you cannot separate their slowness from yours (Calling Something You Do Not Control).
Label with bounded values only: route *pattern*, method, status class, dependency name, outcome. Never user id, tenant id, order id, full path or raw error message.
Use histograms, not averages, for anything time-shaped, and choose bucket boundaries that bracket the latencies you care about — the buckets decide which percentiles are meaningful (Histograms: A Distribution You Can Afford to Keep Forever).
Emit business counters too: orders placed, payments captured, signups. A drop in a business metric detects outages that every technical metric misses.
Give every metric a name and unit convention and stick to it (_total for counters, _seconds for durations). Renaming a metric later breaks every dashboard and alert built on it.

What can go wrong

Failure modes

Cardinality explosion: adding a user id or a raw URL path as a label multiplies series until the metrics backend degrades or starts dropping data — and the outage is in your observability system during an incident.
Counting errors only where they are caught, so absorbed failures never appear and the error rate says everything is fine while callers see degradation.
Averaging latency, or averaging pre-computed percentiles across instances, which produces a number that is not any percentile of anything.
A gauge scraped at intervals missing a spike entirely — a pool that saturates for eight seconds between two thirty-second scrapes leaves no trace.
The mitigation failing: an alert on a metric that stops being emitted. A missing series is not a healthy series, and "no data" must alert too.
Instrumenting the middleware but not the paths that bypass it — static routes, health endpoints, error paths that return early — so the request rate under-reports.

What can race

A gauge updated from multiple concurrent requests must be incremented atomically. An in-flight counter maintained with a non-atomic read-modify-write drifts and eventually reads negative (Atomic Operations).
Counter increments race with scrapes. This is benign for rates — a scrape sees a slightly stale value — but it means a single scrape is never an exact instant.

Security

A metrics endpoint is an information disclosure surface. Series names reveal internal structure, route names reveal your API, and business counters reveal transaction volumes. Do not expose it publicly (Public Exposure, Read With Context).
Never put identifiers in labels — beyond cardinality, a tenant id in a label makes per-customer volumes readable by anyone with dashboard access (Multi-Tenancy).
Metrics can leak through timing even without labels: a per-route latency histogram on a login endpoint can expose whether an account exists, if the code paths differ enough.
Authorization denials are worth a counter, labelled by rule rather than by principal. A spike in denials is a signal worth alerting on (Audit Logs for Privileged Actions).

Misreads

"CPU and memory are monitoring." They tell you about the process. Almost every backend incident is about queueing for a resource, and CPU is flat during most of them.
"p99 is the worst case." It is the worst case for 99% of requests. If a customer makes 100 calls to render a page, they hit the p99 nearly every time (Tail Latency: Why p50 Being Fine Does Not Help).
"We can compute metrics from logs." You can, expensively and slowly. It works until the volume that makes metrics necessary is the volume that makes log aggregation unaffordable.
"More labels means more insight." More labels means more series. The insight arrives before the cost does, which is why this mistake is always made in good faith.
"The error rate is the 5xx rate." Absorbed failures, degraded responses and timeouts the client gave up on are all invisible in a 5xx count (Partial Failure: When 3 of 5 Succeed).

Operating it

How you see it in production

The two-graph test for any incident: request rate and error rate by route on one, latency percentiles and in-flight count on the other. If those two do not narrow it down, a signal is missing.
Watch pool wait time rather than pool utilisation alone. Utilisation at 100% with zero wait time is a well-sized pool; any wait time at all means requests are queueing for a connection (Connection Pools).
Compare your latency histogram against the sum of your dependency histograms. The unexplained remainder is time spent in your own code or waiting for a runtime resource.
Overlay deploy markers on every dashboard. "It started at 14:02" and "we deployed at 14:01" is the fastest diagnosis there is (Deploys Are the First Suspect).

What changes at 10x and 100x

At 10x, metrics cost barely moves — that is the property that distinguishes them from logs. Cost scales with series count, not with request count.
At 100x, series count is the constraint, and the pressure to add "just one more label" is what breaks metrics systems. Cardinality discipline is the scaling story here.
With many instances, per-instance gauges must be aggregated correctly: sum in-flight counts, sum pool usage, but merge histogram buckets rather than averaging percentiles.
Above a certain fleet size, scrape-based collection has its own load. Push-based or aggregation-tier designs appear — a tooling concern, but one that changes what your service must emit.

What this costs

Every metric is a permanent cost: it is stored forever, at every scrape, whether or not anyone looks at it. Unused dashboards are still paid for.
Metrics cannot answer "what happened to this request". They aggregate by construction, so they must be paired with logs and traces rather than replacing them.
Histogram buckets are chosen in advance. Choose badly and your p99 is uninformative, and fixing it means changing the metric and losing comparability with history.
Rich labels are exactly what makes a dashboard useful and exactly what makes the system expensive. There is no configuration that gives both.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe eight signals are tool-independent. Every metrics system can express counters, gauges and histograms; the names and the query language differ.
FRAMEWORK-SPECIFICPrometheus is used as the example here: it is pull-based, computes rates from monotonic counters at query time, and stores one series per label combination. StatsD is push-based and pre-aggregates, so counters are already rates and percentiles are computed per-agent — which makes cross-instance percentile aggregation wrong in a way Prometheus histograms avoid. OpenTelemetry metrics abstract over both and add a delta-versus-cumulative distinction (OpenTelemetry Concepts).
RUNTIME-SPECIFICThe saturation signal that matters most depends on the runtime: event-loop lag on Node, GIL contention and worker-process busy count on CPython, thread-pool queue depth on the JVM, goroutine count and scheduler latency in Go. "In-flight requests" is universal; the resource that saturates first is not (Backend Runtime Models).

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.