OpenTelemetry Concepts
A vendor-neutral way to describe traces, metrics and logs, propagate context across process boundaries, and ship the result anywhere. Worth understanding as a set of concepts — signals, context, semantic conventions, collector — rather than as a product to install.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Signals, context, and the pieces that matter
The conceptual model is small. Signals are traces, metrics and logs, with profiles as a newer addition. A span is one operation with a start, a duration, a status and attributes; spans form a tree within a trace. Context is the small piece of data — trace id, span id, flags — that must travel with a request so a downstream span can attach to the right parent. Propagators serialize that context into and out of carriers such as HTTP headers or message metadata.
The SDK implements emission in each language; the collector is a separate process that receives, processes and exports telemetry; exporters speak the backend's protocol. The design intent is that application code depends only on the API, so changing backends is a collector configuration change rather than a code change across every service.
Semantic conventions are the piece that is easy to skip and expensive to skip. They are agreed attribute names — for HTTP method, route, status, database system, messaging destination — so a query works the same across a Python service and a Go service. Without them, each service invents http.status, httpStatus and status_code, and cross-service queries silently return partial results.
Context propagation is where it succeeds or fails
Almost every disappointing tracing rollout has the same root cause: context stops propagating somewhere. HTTP is usually fine, because auto-instrumentation handles header injection and extraction. The breaks happen at boundaries the libraries do not cover — a message published to a queue without context in its metadata, a background job scheduled from a request, a thread pool that loses the ambient context, a manually constructed HTTP client.
The symptom is distinctive and worth recognizing: traces that end abruptly at a service boundary, and orphan traces that begin at a consumer with no parent. If a trace waterfall shows the request "finishing" while the work demonstrably continues elsewhere, context propagation is broken at that hop, not the instrumentation (see Carrying the Trace Across the Gap).
The async boundary deserves particular attention because it is both the most commonly broken and the most valuable to fix. A request that enqueues work and returns has its real latency *after* the response, in the worker; without propagation, the two halves are unlinkable and end-to-end latency for the user-visible outcome is unmeasurable (see Six Queue Signals, Two That Wake You Up).
1// HTTP: handled automatically by auto-instrumentation.2// Queues and background jobs: usually NOT. Context must be carried by hand.3 4// Producer — inject the active context into the message metadata5function publish(topic: string, body: unknown) {6 const headers: Record<string, string> = {}7 propagation.inject(context.active(), headers) // adds traceparent8 return broker.publish(topic, { body, headers })9}10 11// Consumer — extract it and make the worker span a child of the producer span12async function onMessage(msg: Message) {13 const parent = propagation.extract(context.active(), msg.headers)14 await context.with(parent, async () => {15 const span = tracer.startSpan('worker.process', {16 // 'links' when the work is causally related but not synchronously nested17 attributes: { 'messaging.destination': msg.topic },18 })19 try {20 await handle(msg.body)21 } finally {22 span.end()23 }24 })25}26 27// Without inject/extract: the producer trace ends at publish, the worker28// starts an orphan trace, and end-to-end latency is unmeasurable.What it buys, and what it costs
The honest case for a vendor-neutral standard is portability and consistency: one instrumentation vocabulary across languages, and backend changes that do not require touching application code. For an organization with several languages and a backend it may want to change, that is substantial. For a single-language team happy with its current vendor, the benefit is smaller and the migration cost is real.
The costs are worth stating plainly. There is a learning curve around context, propagators and the SDK/collector split. The collector is another production component to run, scale and monitor — one that, if it falls over, takes observability with it exactly when it is needed. And the specification evolves: signal stability differs per signal and per language, so a team should check current status rather than assume uniformity across the project.
The concepts, though, outlive any particular implementation. Spans and traces, context propagation across process boundaries, semantic conventions, and a processing layer between application and backend are the durable ideas. A team that understands those can evaluate any tracing stack — and can recognize the failure modes above regardless of which library produced them.
| Situation | Case for | Case against |
|---|---|---|
| Several languages in the request path | One vocabulary and one propagation format across all of them | Per-language SDK maturity differs; check the ones you actually use |
| Might change observability vendor | Backend swap becomes collector configuration, not a code migration | The abstraction has its own learning and operational cost |
| Single language, single vendor, happy | Consistency with the wider ecosystem; conventions are still useful | Migration cost is real and immediate; the benefit is optionality |
| Small team, one service | Semantic conventions are worth borrowing regardless | A collector is another component to run for limited gain |
| Need redaction or tail sampling centrally | The collector is the natural place for both | It becomes a stateful, capacity-planned dependency |
Key points
- The model is small: signals (traces, metrics, logs), spans in a trace, context that propagates, and a collector between application and backend.
- Semantic conventions — agreed attribute names — are what make cross-language, cross-service queries actually work.
- Most disappointing rollouts fail at context propagation, usually at queue and background-job boundaries rather than HTTP.
- A trace that ends at a service boundary, or an orphan trace starting at a consumer, is the recognizable signature of broken propagation.
- The collector buys central sampling and redaction, and costs you another production component that can take observability down with it.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request → service A: a span is created and context is injected into outbound HTTP headers automatically.
- 2Service A → queue: the message is published without context in its metadata, because no auto-instrumentation covers this client.
- 3Queue → worker: the consumer starts a new root span, so the causal link to the originating request is lost.
- 4Worker → trace backend: two unrelated traces are stored where one was expected, and end-to-end latency cannot be computed.
- 5Engineer → dashboard: the request appears to finish at publish time, hiding the several seconds of work that follow.
- • "Every service has tracing, so we have distributed tracing." Six disconnected traces per request are six local traces, not a distributed one.
- • "The trace shows the request took 40ms." If context stops at an async boundary, the trace shows the synchronous portion only.
- • "Adopting a standard means we are vendor-neutral." Neutral at the instrumentation layer; dashboards, alerts and queries still tend to be backend-specific.
- • "The specification is stable." Stability differs per signal and per language implementation; check the status for the ones you actually depend on.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Count the fraction of traces that span the full request path versus those ending at a boundary — the gap identifies where propagation breaks.
- • Check attribute-name consistency across services for the same concept; divergence means semantic conventions are not being applied.
- • Monitor collector health as a production dependency: queue depth, refused batches, export failures, memory.
- • Verify async linkage explicitly: for a request that enqueues work, confirm the worker span shares the originating trace id.
- • Instrument context propagation explicitly at every async boundary: queue publish/consume, background job scheduling, thread and worker pools.
- • Adopt semantic conventions for common attributes so cross-service queries work without per-service translation.
- • Run a collector for central sampling, redaction and export, and treat it as a monitored production dependency.
- • Keep application code on the API rather than a vendor SDK, so backend changes stay configuration changes.
- • Add a trace-continuity check to integration tests: an end-to-end request should produce one connected trace.
- • Send a synthetic request through the full path and confirm one trace contains spans from every service, including async workers.
- • Query by a semantic attribute across two services in different languages; the result should include both without special-casing.
- • Confirm collector metrics show zero refused or dropped batches at peak load.
- • The collector is an additional production component with its own capacity planning, failure modes and upgrade cycle.
- • Vendor neutrality at the instrumentation layer does not extend to dashboards, alert definitions or query languages.
- • Manual propagation at async boundaries is code that must be written and maintained in every service that publishes work.
- • Add trace-continuity assertions to integration tests so a new async boundary cannot silently break propagation.
- • Alert on orphan-trace rate — traces starting at a consumer with no parent — as a propagation regression signal.
- • Pin and review SDK and convention versions deliberately; silent attribute renames break saved queries and alerts.
Accuracy
Performance numbers are conditional. These are the conditions.
- ENVIRONMENT-SPECIFICSignal stability, auto-instrumentation coverage and SDK maturity vary by language and by release. Check the current status for the languages you use rather than assuming uniformity.
- ILLUSTRATIVEThe code examples show the shape of inject/extract rather than the exact API of any specific SDK version; names and signatures differ between languages and releases.