ObservabilityGENERALRUNTIME-SPECIFICPROTOCOL-SPECIFIC

Correlation Ids That Survive Every Hop

One identifier that follows a request through services, queues and workers — including the hop into a background job, which is where it is usually dropped.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

A customer reports a failure at 14:32. How do I find every log line, in every service, that belongs to that one request?

The requirement

A user says checkout failed. The request touched the API, the inventory service, a queue, a worker and a payment provider. Right now finding the related log lines means five greps and a lot of guessing.

The obvious build

Log a timestamp and the user id everywhere. If we need to find a request, we filter by user and narrow the time window.

Why it breaks

A user with concurrent requests — two tabs, a retry, a mobile app polling — produces interleaved lines that cannot be separated by user and time.

How it breaks in production

A user with concurrent requests — two tabs, a retry, a mobile app polling — produces interleaved lines that cannot be separated by user and time.
Clocks differ between hosts. The downstream service's log for a request can appear before the upstream's, so time-window filtering silently excludes the lines you need.
The background job runs minutes later, on another host, with no user in scope. It is outside every window you would think to search.
Anonymous requests have no user id at all, and those include the signup and login failures you most want to debug.
Even when it works it is manual. The question "show me this request" should be one query, not an investigation.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

A correlation id is a single opaque value, generated once at the outermost boundary, attached to every log line, span, outbound request and enqueued message produced while handling that request.
It has to be generated at the edge — the load balancer, gateway or first middleware — because anything generated deeper misses whatever happened before it (The Middleware Pipeline).
It must be accepted from a trusted caller and generated otherwise. An upstream service passing X-Request-Id should keep its id so the two logs join; an anonymous internet client's header must not be trusted blindly.
It propagates in-process via whatever the runtime offers for request-scoped ambient state: AsyncLocalStorage in Node, contextvars in Python, an explicit context.Context in Go, a thread-local plus MDC on the JVM (Request Context Propagation).
It propagates across processes as a header on outbound HTTP, as metadata on a gRPC call, and — the step people forget — as a field on the job payload when work is enqueued.
The hop into a background job is different in kind: there is no request in scope on the worker side, so the id has to be *data in the message* and then re-established as ambient context at the top of the job handler.
A correlation id is related to but not the same as a trace id. A trace id identifies one distributed trace with sampling attached; a correlation id is a business-level thread that should never be sampled away (Tracing From the Backend's Side).

One id, five processes

The value of a correlation id is entirely in how far it travels. An id that covers only the API process answers a question you could already answer. The interesting hops are the ones where the request stops being a request: an enqueue, a scheduled retry, a fan-out into ten child jobs.

In the flow below, every arrow is a place the id must be explicitly carried. Four of them are HTTP headers, which most frameworks or a shared client will handle. The one into the queue is data in the message envelope, and it is the one that breaks.

UserLLMAgentToolDataDecisionHumanGuardrail

The hop into a background job

GENERALThe envelope pattern is queue-agnostic. What differs is where metadata belongs natively: SQS and Kafka have message attributes/headers, so the envelope can live there instead of in the body; a plain Redis list has no such concept and the envelope must be part of the serialized value.

A worker process has no request. Whatever ambient context existed died when the enqueuing request returned, possibly minutes ago and certainly on a different machine. The only thing that crosses is the message, so the id has to be in the message.

Put it in an envelope rather than in the business payload. The payload is the domain object, versioned and validated; the envelope is transport metadata — correlation id, enqueue time, attempt count, schema version. Mixing them means the correlation id shows up in your domain types forever.

Enqueue with an envelope; restore context on the other side

1// --- Producer side, inside a request -------------------------------
2async function enqueue<T>(topic: string, payload: T) {
3  await queue.send(topic, {
4    meta: {
5      correlationId: context.get('correlationId'),  // ambient, set at the edge
6      enqueuedAt: new Date().toISOString(),
7      schema: 1,
8    },
9    payload,
10  })
11}
12 
13// --- Consumer side, no request in sight ----------------------------
14queue.subscribe(topic, async (msg) => {
15  // Re-establish ambient context FIRST, before any logging or work.
16  await context.run({ correlationId: msg.meta.correlationId ?? randomUUID() }, async () => {
17    logger.info({ topic, attempt: msg.attempt }, 'job started')
18    await handle(msg.payload)
19  })
20})
21 
22// A job that fans out MUST copy the id down, or generation two is orphaned.
23async function fanOut(items: Item[]) {
24  for (const item of items) await enqueue('item.process', item)  // enqueue() reads ambient ctx
25}

The ?? randomUUID() matters: a message with no id still gets one, so the worker's lines are internally joinable even when the chain to the origin is broken. Log the fallback separately so you can find the producer that omitted it.

Correlation id, trace id, job id

These three get conflated constantly, and the conflation causes real gaps. They have different lifetimes, different cardinalities and — critically — different sampling behaviour.

The rule of thumb: correlation ids are for humans reading logs during an incident and must never be sampled; trace ids are for tools computing latency breakdowns and are usually sampled; job ids are for retry bookkeeping and mean nothing outside the queue.

Identifier	Spans	Sampled?	Answers
Correlation id	The whole causal chain, including jobs triggered days later	Never	"Show me everything caused by that one click"
Trace id (W3C `traceparent`)	One distributed trace, hop by hop, with parent/child spans	Usually — head or tail sampled	"Where did the 3 seconds go?" (Trace, Span, Attribute, Status)
Span id	One operation inside a trace	With its trace	"Which call inside this hop was slow?" (Parents, Children and Links)
Job id	One queued message and its retries	Never, but scoped to the queue	"Did this job run, and how many attempts?" (Job Queues)
User / tenant id	All activity by a principal	Never	"Is this customer affected?" (Multi-Tenancy)

How to build it

Most important first.

Generate at the edge if absent; accept an inbound id only from callers you authenticate. Use a UUIDv4 or a ULID — something with no meaning and no collision risk.
Put it in ambient request-scoped context so no function signature has to carry it, then make the logger read it automatically. If engineers must remember to pass it, it will be missing exactly where it matters.
Return it in the response, both as a header and inside error bodies, so a caller can quote it (Not Leaking Your Internals).
Attach it to every outbound call — a wrapped HTTP client that sets the header, not a per-call-site argument.
Add it to the job envelope when enqueuing, and restore it into context as the first thing a worker does. Envelope, not payload: it is metadata about the message, not part of the business data (Job Queues).
Carry a second id where it helps: keep the original correlation_id across the whole causal chain, and add a per-hop request_id if you need to distinguish a retry from its original.
Never let it be optional in log output. A logger that omits the field when context is empty makes the gap invisible.

What can go wrong

Failure modes

The id lost at an async boundary: a setTimeout, a detached promise, a thread handed to a pool, or a library that does not propagate context. The lines after that point have no id and look like they belong to nothing.
A worker that logs the job id but not the correlation id, so the chain breaks precisely at the hop you added a queue for.
Fan-out losing the association: a job that enqueues ten children without copying the id, so the tenth generation is unreachable from the original request.
The mitigation failing — a middleware that generates the id *after* the logging middleware runs, so the access log line for every request has no id.
Two different header names in use (X-Request-Id, X-Correlation-Id, traceparent), each honoured by a different service, producing three disjoint views of one request.
A retried job generating a fresh id, so the failure and its successful retry cannot be connected.

What can race

Concurrent requests in one process share the same logger and often the same module-level state. Ambient context is what keeps their ids separate — a correlation id stored in a module variable will be overwritten by the next request mid-flight (Backend Races).
A job that starts before the enqueuing transaction commits can log a correlation id whose originating request later rolls back, producing a trail for something that never happened (The Transactional Outbox).

Security

A client-supplied id is untrusted input. Bound its length and character set before it reaches a log — an unbounded header becomes log injection or a log-storage cost attack (Every Input Surface).
Never derive the id from anything meaningful. A correlation id containing a user id, an email or a tenant name leaks that into every downstream system and into caller-visible responses.
Do not reuse a session token or an API key as a correlation id, however convenient the uniqueness is. It ends up in logs, in third-party observability tools and in error bodies (Secrets in Logs).
Accepting an inbound id from an unauthenticated caller lets them join their own traffic to yours in your logs, and lets them forge collisions with a known id. Accept from trusted peers, generate otherwise.

Misreads

"We have tracing, so we do not need correlation ids." Traces are sampled and usually short-lived. A support ticket arrives three days later about a request that was not sampled.
"The job id is enough." It identifies the job, not what caused it. The question during an incident is almost always "what request led to this", and the job id cannot answer it.
"Propagation is automatic — the framework does it." It does it for the paths the framework controls. Your queue client, your custom thread pool and your third-party SDK are not those paths.
"One id is enough for everything." A correlation id spans the causal chain; a trace id spans a sampled trace; a job id spans one execution. They overlap and are not substitutes.

Operating it

How you see it in production

The test that matters: pick a random production correlation id and confirm you can retrieve API, service, worker and dependency lines with a single query.
Graph the percentage of log lines that carry a correlation id, per service. Any value below 100% for request-scoped services names a propagation gap.
Alert on worker log lines with no correlation id. That number should be near zero, and its rise is always a regression in the enqueue path.
Include the id in outbound calls so a dependency's support team can find your request in *their* logs — this is often the fastest path through a third-party incident.

What changes at 10x and 100x

At 10x, the id is what makes log search remain usable: filtering a large index by an exact high-cardinality field is cheap, while filtering by user plus a time range is not.
At 100x, sampling arrives. Sample traces if you must, but keep correlation ids on all logs — sampled-away ids are exactly the ones a customer is asking about (Sampling Without Throwing Away the Evidence).
Across many services, the propagation contract must be a shared library or a platform concern. Per-team implementations diverge on header name and on which async boundaries they cover.

What this costs

Every log line grows by the size of the id, and it is a high-cardinality field in your log index. That is storage and indexing cost, paid continuously for value you only collect during incidents (The Log Bill and What It Is Buying).
Ambient context has a runtime cost and, in some runtimes, sharp edges — Node's AsyncLocalStorage has measurable overhead and libraries that break the async chain silently drop it.
Explicit passing is more reliable and pollutes every signature. Most teams take ambient context and accept the occasional gap; both choices are defensible and neither is free.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe pattern is stack-independent; only the ambient-context mechanism changes.
RUNTIME-SPECIFICNode uses AsyncLocalStorage, which follows async/await and most promise chains but is lost across some native callbacks and worker threads. Python's contextvars propagate into asyncio tasks but not into a ThreadPoolExecutor without explicit copying. Go has no ambient context at all by design — ctx is a parameter, which never silently drops but does mean every signature carries it. The JVM uses thread-locals plus MDC, which break when work moves to another thread unless the executor is instrumented.
PROTOCOL-SPECIFICW3C Trace Context defines traceparent and tracestate and is what OpenTelemetry propagates; X-Request-Id is a convention, not a standard. Carrying both is common: the standard header for tracing tools, the conventional one for humans and logs.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designrequest-ids

Architecturedistributed-tracing

Performancecorrelation-ids context-propagation distributed-tracing trace-sampling opentelemetry structured-logging

Concurrencyreasoning-about-races

Domains that do not exist yet

— Distributed Systems — causality tracking beyond a single identifier: happens-before, vector clocks, and why "the whole causal chain" is harder than one propagated string suggests.