Instrumentation: From Code to Signal

A metric on a screen has travelled a long way. Application code calls an instrumentation API. The SDK aggregates in process — incrementing a counter, adding to a histogram bucket, buffering a span. A background exporter batches and ships on an interval. A collector receives, optionally processes (sampling, filtering, redacting, enriching) and forwards. A backend ingests, indexes and stores. A query engine reads it back when someone opens a dashboard.

Every hop is a place data can be lost, and most losses are silent by design — telemetry that blocks the application is worse than telemetry that drops. When the exporter queue fills, spans are dropped. When the collector is overloaded, batches are rejected. When ingestion is rate-limited, data arrives late or not at all. None of that produces an error in your application; it produces a chart that looks calm.

This is why pipeline health signals belong on the dashboard next to the service signals. "Spans dropped per second" and "collector queue depth" are the difference between "the incident resolved at 14:20" and "the telemetry stopped at 14:20". During a real outage those are the same picture, and they mean opposite things.

Automatic instrumentation hooks known libraries — the HTTP server, the database driver, the HTTP client, the queue consumer — and produces spans and metrics without application code changes. It is the fastest route to a useful trace and it covers exactly the boundaries that matter most: the ones where a process talks to something else.

What it cannot know is your domain. It will tell you POST /checkout took 1.8s and that a payments HTTP call took 1.6s of it. It will not tell you the request was for a high-value cart, in the EU, on the new pricing path, retried twice. Those attributes come from manual instrumentation, and they are what makes a trace answer questions later (see Observability Is Not a Dashboard).

The practical pattern is: automatic for coverage, manual for meaning. Turn on auto-instrumentation for framework and client boundaries, then add a small number of high-value attributes and a few spans around domain operations that have no library boundary — a pricing calculation, a rules evaluation, a batch of work inside one handler. Adding a span per function is a common overreaction that produces expensive noise.

Instrumentation is not free and pretending otherwise leads to the second-order incident where telemetry causes the outage. There are four distinct costs: in-process CPU and allocation to record data, memory to buffer it, network egress to ship it, and storage plus query cost at the backend. They scale differently — CPU with call frequency, storage with cardinality and retention — so the same change can be cheap in one dimension and ruinous in another.

The classic self-inflicted failure is a high-cardinality metric label. Adding user_id to a counter with 50,000 users does not add one series; it multiplies existing series by 50,000. Metrics backends fall over from this regularly, and the failure often takes down monitoring for *other* teams sharing the backend (see Cardinality: The Label That Took Down Monitoring). The corresponding trace failure is logging full request bodies as span attributes: cheap in CPU, brutal in storage, and a data-protection problem on top.

Sampling is the main lever, and it belongs at different places for different signals. Head sampling decides at the start of a request and is cheap but blind — it drops slow requests as readily as fast ones. Tail-based sampling decides after the trace completes, keeping errors and slow traces, at the cost of buffering every trace in the collector (see Sampling Without Throwing Away the Evidence). For logs, level and rate limits are the equivalent lever (see The Log Bill and What It Is Buying).