Label Sets That Survive a Year
A label set is a schema: bounded value sets, names that mean the same thing in every service, and a migration path for the day you need to change one. Get it wrong and you either cannot join across services or cannot afford the series you created.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Templates, not values
The single most common unbounded label is the raw request path. /users/8412/orders/99123 is a distinct label value from /users/8413/orders/99124, so a metric labelled by raw path has cardinality equal to the number of distinct URLs your users visit — effectively unbounded, and growing forever (see Cardinality: The Label That Took Down Monitoring).
The fix is the route template: the pattern the router matched, /users/{id}/orders/{orderId}, which has exactly as many values as you have routes. It is also the more useful label, because "how slow is the order-detail endpoint" is the question anyone actually asks, and per-URL breakdown answers nothing that a trace search would not answer better.
The same normalization applies elsewhere. Status classes (2xx, 4xx, 5xx) rather than exact codes when you only branch on the class; error *types* from a closed enum rather than exception messages, which are free-form strings and therefore unbounded; and instance identity only where you genuinely compare instances, since it multiplies everything by the fleet size.
| Tempting label | Cardinality | Use instead | What you keep |
|---|---|---|---|
path="/users/8412" | Unbounded — one per URL visited | route="/users/{id}" | Per-endpoint latency, which is the question people ask |
status="503" | Bounded but wide, and rarely branched on exactly | status_class="5xx" | Error-ratio SLIs; keep exact codes in logs |
error=exc.message | Unbounded — messages contain ids and values | error_type="timeout" from a closed enum | Error breakdown by kind; the message lives in the log |
customer_id=... | Unbounded and permanent | span attribute + log field | Per-customer debugging, at per-event cost |
instance=pod-7f3a | Fleet size, multiplying every other label | Only on resource metrics where you compare instances | Instance comparison where it matters, without multiplying request metrics |
Consistency is what makes fleet queries possible
A label name is an API between services, and like any API its value comes from everyone agreeing. If checkout emits service="checkout" and search emits svc="search", then no single query aggregates across both, and every fleet-wide dashboard becomes a union of special cases that silently omits whichever service was added last.
The cheapest fix is a shared instrumentation library that attaches the standard labels — service, environment, region, version — automatically, so individual services cannot get them wrong. OpenTelemetry's semantic conventions exist for exactly this reason and are worth adopting even if you never adopt anything else from it (see OpenTelemetry Concepts).
The version label deserves specific mention: it is what makes "What Changed?" — Deploy Markers and the Invisible Deploys and regression comparison work, letting you split any metric by build and see the two populations side by side during a rollout. It is bounded in practice as long as old versions stop reporting, and it pays for itself the first time a canary is worse than the baseline.
1# One place, applied to every metric the service emits.2standard_labels = {3 "service": SERVICE_NAME, # bounded: one per service4 "environment": ENV, # prod | staging | dev5 "region": REGION, # eu | us | ap6 "version": BUILD_SHA[:7], # bounded: old versions stop reporting7}8 9# Per-metric labels are added on top, and must also be bounded:10requests.labels(**standard_labels, route=route_template(req),11 status_class=status_class(res)).inc()12 13# Now a fleet-wide query works without special cases:14# sum by (service) (rate(http_requests_total{status_class="5xx"}[5m]))15# / sum by (service) (rate(http_requests_total[5m]))16#17# And a canary comparison is one extra grouping:18# histogram_quantile(0.99, sum by (version, le) (...))Renaming a metric is a migration
Metric names and labels are consumed by dashboards, alert rules, recording rules, SLO definitions and runbooks — most of which are not in the service's repository and will not be updated by the change that renames the metric. Renaming in place therefore breaks alerting silently: the rule keeps evaluating, finds no series, and returns no data. Depending on the backend, "no data" is either a permanent non-alert or a permanent alert, and both are bad.
The safe path is the same dual-write pattern used for any contract change (compare API Migration: Running the Change End to End in API Design): emit both the old and new names for a full retention period, migrate consumers, verify nothing still reads the old series, then remove it. The cost is temporary duplicate cardinality, which is exactly why you want to have gotten the label set right the first time.
Historical comparability is the part people forget. Even a perfect migration leaves you with a discontinuity: queries spanning the cutover must union two series. If the change also altered bucket boundaries or label semantics, the two sides are not directly comparable at all, and any "we improved p99 by 30%" claim across that boundary needs re-checking (see Regression or Tuesday? Telling a Real Change from Noise).
phase 1 emit both, change nothing else
http_request_duration_seconds{path=...} # old, still emitted
http_server_duration_seconds{route=...} # new, semantic-convention name
cost: duplicate series for the duration of the migration
phase 2 migrate consumers, one at a time
[x] service dashboards
[x] alert rules + recording rules
[ ] SLO definitions <- not in this repo
[ ] on-call runbook queries <- not in any repo
phase 3 verify nothing reads the old name
query the backend for reads against the old series
wait one full retention period so historical queries still work
phase 4 stop emitting the old name
skipping to phase 4 directly:
alert rules evaluate against zero series -> no data
-> alerts that can never fire, discovered during the next outageKey points
- Use route templates, status classes and closed error enums; raw paths, messages and ids are unbounded label values.
- Label names are an inter-service API — inconsistent names make fleet-wide aggregation impossible without special cases.
- Attach standard labels (service, environment, region, version) from shared instrumentation so services cannot diverge.
- A
versionlabel is what makes canary comparison and deploy-correlated regression analysis possible. - Renaming a metric or label is a dual-emit migration, because most consumers live outside the service repository.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Team → instrumentation: each service picks its own label names, and each service's own dashboard works fine.
- 2Instrumentation → backend: series carry inconsistent label keys for identical concepts.
- 3Backend → query: a fleet-wide SLO query cannot group across services without a per-service special case.
- 4Query → dashboard: the aggregate dashboard silently omits services whose labels do not match the query.
- 5Dashboard → SLO: the reported fleet error ratio is computed over a subset of services, and nobody notices until an omitted service has an outage that never appears.
- • "Our dashboards work, so the labels are fine" — per-service dashboards work with any naming; only aggregation exposes the inconsistency.
- • "The fleet SLO looks healthy" — check which services are actually included before trusting an aggregate that silently drops non-matching series.
- • "Renaming is safe, the code compiles" — dashboards, alert rules and runbooks are not compiled and will not fail visibly.
- • "Exact status codes are more precise, so they are better labels" — precision you never query for is cardinality you always pay for.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • List every distinct label name in use across services and look for synonyms (`svc`/`service`, `path`/`route`/`endpoint`).
- • Check each label's distinct value count against its expected bound; anything growing linearly with users or requests is unbounded.
- • Before a rename, query the backend for which alert rules, recording rules and dashboards reference the old series.
- • Split a key latency metric by `version` during a rollout to confirm the label is present and populated.
- • Adopt a shared instrumentation wrapper that attaches standard labels automatically and normalizes routes to templates.
- • Follow semantic conventions for names so third-party tooling and future engineers agree with you by default.
- • Move unbounded breakdowns to span attributes and log fields, keeping metric labels bounded (see [[cardinality]]).
- • Run renames as dual-emit migrations with an explicit consumer checklist and a full retention period of overlap.
- • Run a fleet-wide aggregation query and confirm the number of services in the result equals the number deployed.
- • Confirm each label's series count matches the hand-computed bound after the change.
- • During a rename, verify the new series is populated and the old one has zero readers before removing it.
- • Route templates lose per-URL detail, so pathological single-URL problems must be found through traces instead.
- • Shared instrumentation libraries are another dependency to version and roll out across every service.
- • Dual-emit migrations temporarily double cardinality for the migrated metric, which is real cost during the overlap.
- • CI check: fail the build when a metric emits a label value outside its declared allowlist.
- • Alert on "no data" for critical alert rules, so a rename that orphans a rule pages someone instead of going quiet.
- • Require a label-set review for any new metric, with expected cardinality stated in the pull request.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe label names, migration phases and query snippets are teaching examples. Real semantic conventions specify names precisely and are worth following over anything invented here.
- ENVIRONMENT-SPECIFICHow a backend handles "no data" in an alert rule — never fires, always fires, or fires a distinct no-data alert — differs by system and changes how dangerous a silent rename is.