Cardinality: The Label That Took Down Monitoring
Cardinality is the product of every label's distinct value count, and it multiplies. One user_id label turns a three-series metric into three million, and the first thing that breaks is the monitoring system you were relying on to tell you what broke.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Cardinality multiplies, and the multiplication is invisible in code
A time series is uniquely identified by the metric name plus the full set of label values. http_requests_total{route="/checkout", status="200", region="eu"} is one series; change any label value and it is a different series with its own storage, its own memory in the ingester, and its own index entries.
The total is the product of the distinct values of each label, not the sum. Four routes, five status classes and three regions is 4 x 5 x 3 = 60 series — completely fine. Add a customer_id label with 50,000 values and it becomes 3,000,000, and nothing in the source code looks any different. The line that added it is one string.
Then multiply by bucket count if it is a histogram. A ten-bucket histogram with that same label set is 30 million series from a single instrumentation call. This is the mechanism behind essentially every "our monitoring fell over" story, and it is why Label Sets That Survive a Year treats label design as a bounded-set problem rather than a naming problem.
metric: http_requests_total
BEFORE
route 4 values (/checkout, /cart, /search, /account)
status 5 values (2xx, 3xx, 4xx, 5xx, timeout)
region 3 values (eu, us, ap)
4 x 5 x 3 = 60 series
AFTER (+ customer_id, 50,000 active customers)
4 x 5 x 3 x 50,000 = 3,000,000 series
as a histogram with 10 buckets
3,000,000 x 10 = 30,000,000 series
rough ingester memory at ~2 KB/series (backend-dependent)
30,000,000 x 2 KB = ~60 GB
the code change that caused this:
- requests.labels(route, status, region).inc()
+ requests.labels(route, status, region, customer_id).inc()What actually breaks, and in what order
The failure is rarely a clean error message. Ingestion memory grows first, because most backends hold an in-memory index of active series. Queries slow down next, since every aggregation has to scan a far larger index — and dashboards that touch the affected metric start timing out while unrelated dashboards still work, which makes the cause hard to spot.
Then retention degrades: either the backend starts dropping samples, or someone shortens retention to fit the disk, and you lose the historical baseline you needed to tell whether today is abnormal. Finally the ingester restarts under memory pressure, which drops in-flight data and resets counters across the fleet (see Counters: The Slope Is the Signal).
The compounding cruelty is timing. High cardinality is usually introduced by a well-intentioned change ("let us break this down per customer so we can debug faster"), and it degrades the monitoring system precisely when load is high — which is when you most need it. The postmortem for the original incident ends up containing a second incident.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Backend active series | 31.2M (was 890k) | A 35x jump — this is the cause, not a consequence | smoking gun |
| Series by metric name | `http_request_duration`: 30.1M of the total | One metric accounts for 96% — check its label set and its recent diff | smoking gun |
| Ingester resident memory | 58 GB, climbing ~4 GB/hour | On a trajectory to OOM; restarts will drop data and reset counters | suspect |
| Dashboard query p99 | 31s (was 1.2s) | Panels touching that metric time out; unrelated panels are fine | suspect |
| Application CPU / latency | unchanged | The app is fine — cardinality cost lands on the storage side, which is why nobody notices at review time | normal |
Where high-cardinality data actually belongs
The information is not worthless — it is in the wrong signal type. "Which customer was affected" is a per-event question, and per-event questions belong in traces and logs, where the cost model is per-event and sampling is available (see Metrics, Logs, Traces, Profiles). Metrics are for aggregates whose cost must stay flat as traffic grows.
So the rule is: labels answer "which bucket", traces and logs answer "which one". Keep the customer id as a span attribute and a log field, where you can find it by searching; keep the metric labelled by route and status, where you can graph it. You lose nothing except the ability to graph per-customer latency, which is a dashboard almost nobody looks at and everybody pays for.
When per-entity aggregate views are genuinely required — a per-tenant SLO dashboard for a hundred enterprise customers, say — bound the set explicitly. Label only tenants above a size threshold and bucket the rest as other, or emit the per-tenant series from a separate, deliberately-scoped metric with a documented ceiling.
1requests.labels(2 route = req.path, # raw path: /users/8412 -> unbounded3 customer_id = req.customer, # 50,000 values and growing4 user_agent = req.ua, # effectively unbounded5).inc()6 7# every new customer permanently adds series8# every UA string variant permanently adds series9# nothing in review flags it; the app is unaffected1requests.labels(2 route = route_template(req), # "/users/{id}" -- 4 values3 status = status_class(res), # 5 values4 region = REGION, # 3 values5).inc()6 7span.set_attribute("customer.id", req.customer) # searchable, per-event cost8log.info("request", customer_id=req.customer, route=..., duration_ms=...)9 10# metric series stay at 60; "which customer" is answered by11# querying traces/logs, which is where per-event questions belongBoth versions can answer "which customer was slow". Only the second keeps that answer in a signal whose cost is per-event and samplable, instead of in one whose cost is permanent and multiplied by every other label.
Key points
- Cardinality is the product of distinct label values, multiplied again by bucket count for histograms.
- Adding one unbounded label — user id, request id, email, raw path — can multiply series by four or five orders of magnitude.
- The cost lands on the metrics backend, not the application, so the change looks harmless in review and in local testing.
- Symptoms appear in order: ingester memory, then query latency, then retention loss, then a restart that drops data.
- Per-entity questions belong in traces and logs; metric labels should answer "which bucket", never "which one".
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Change → instrumentation: a
customer_idlabel is added to a histogram to enable per-customer debugging. - 2Instrumentation → backend: series count goes from ~890k to ~31M as customers appear in traffic over the following hours.
- 3Backend → memory: the in-memory series index grows past available RAM and the ingester begins to thrash.
- 4Memory → queries: aggregations scan a vastly larger index; panels touching that metric exceed the dashboard timeout.
- 5Queries → responders: the team loses its observability during peak load, and the metric added to speed up debugging is what removed the ability to debug.
- • "The app is healthy, so the metrics change was fine" — the cost is entirely on the storage side by design.
- • "It worked in staging" — staging has a handful of test customers; cardinality scales with production entity counts, not with code paths.
- • "We can drop the label later" — dropping it stops new growth but the historical series remain until retention expires, and the index stays large.
- • "Only the affected dashboards are slow, so it is a dashboard problem" — selective slowness is the classic signature of one metric with a huge index.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Track total active series on the backend and series count broken down by metric name — the offender is usually one metric.
- • Before merging a label change, multiply out the expected cardinality by hand and write the number in the pull request.
- • Watch ingester memory and dashboard query latency as leading indicators; both move before anything drops.
- • Audit for label values that are ids, emails, URLs with path parameters, user agents, or free-form error strings.
- • Remove unbounded labels immediately and move the identity to span attributes and log fields.
- • Normalize labels that look bounded but are not: route templates instead of raw paths, status classes instead of exact codes.
- • Apply a bounded allowlist where per-entity aggregates are genuinely needed — top-N tenants labelled, everything else as `other`.
- • Configure backend-side cardinality limits and per-metric series caps so the next mistake is rejected rather than absorbed.
- • Confirm active series returns to its previous order of magnitude and stays flat as new customers arrive.
- • Re-run the dashboards that were timing out and compare query latency against the pre-incident baseline.
- • Verify the per-customer question is still answerable via traces or logs, so the capability was moved rather than lost.
- • Bounded labels mean some breakdowns are only available by querying traces or logs, which is slower and often sampled.
- • Backend cardinality limits protect the system by dropping data, so a legitimate new label can be silently rejected.
- • Top-N tenant labelling needs periodic re-evaluation, and the `other` bucket hides exactly the small tenants who complain loudest.
- • Alert on active series count and on per-metric series growth rate, not just on backend memory.
- • Add a CI check that fails when a metric's label set gains a value not on a declared allowlist.
- • Make expected cardinality a required line in any pull request that adds or changes a metric label.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 50,000 customers, 35x series jump and ~2 KB/series memory figure are invented to make the multiplication concrete. Per-series overhead varies substantially by backend and by how many samples are held in memory.
- ENVIRONMENT-SPECIFICSeries limits, index structure and failure behaviour under cardinality pressure differ sharply between backends. Some reject new series, some drop samples, some simply run out of memory.