StrategiesPLATFORM-SPECIFICGENERAL

Shadow Traffic: Real Requests, Discarded Answers

Duplicating production traffic to a candidate that serves nobody — strong evidence about crashes, load and resource use, and no evidence at all about writes.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

How do I run the new version against real production traffic before any user depends on its answers — and what does that actually prove?

The problem

Synthetic load has the shape you imagined. Real traffic has the shape your users actually produce, including the malformed, the enormous, the ancient client and the one endpoint that carries most of the cost. You want the candidate exposed to that before it serves anyone.

What teams do first

Mirror production traffic to the new version and compare. If it handles real traffic without errors, it is ready.

How it breaks

The candidate is connected to the same database. Mirrored writes are real writes: duplicate rows, double decrements, two charges. The mirror is only harmless on paths that do not change state.

How it breaks in production

The candidate is connected to the same database. Mirrored writes are real writes: duplicate rows, double decrements, two charges. The mirror is only harmless on paths that do not change state.
External side effects escape immediately. A mirrored request that sends an email sends a real email, and a mirrored webhook is delivered to a real partner.
Every shared dependency now receives double the read load — the database, the cache, the downstream services — which is a capacity change disguised as a test (Capacity Management).
If the mirroring is synchronous, the candidate's latency becomes the real request's latency, and a slow or failing candidate becomes a production incident.
"No errors in the shadow" says nothing about whether the responses were *correct*, because nobody looked at them.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

A proxy, load balancer or mesh copies incoming requests and sends a duplicate to the candidate. The primary path is unchanged: the user's response comes from the live version, and the candidate's response is discarded or logged.
For the shadow to be safe, mirroring must be fire-and-forget from the primary's point of view: separate connection, own timeout, bounded concurrency, and no ability to delay or fail the real request.
What this buys is the traffic mix. The candidate sees genuine request shapes, genuine payload sizes, genuine header and client diversity and genuine concurrency, without any user depending on what it returns.
Response diffing is the optional second half: capture both responses and compare them. That turns the shadow from a stability test into a behavioural one — but only for endpoints whose responses are deterministic given the same state.
Everything the candidate does beyond returning a response is not shadowed. Writes, publishes, emails, charges and cache mutations happen for real unless something deliberately prevents them.

The shape: one request, two destinations, one answer

The single most important edge in this diagram is the one that is missing: nothing goes from the candidate back to the client. That absence is the safety property, and it is also the reason the evidence is one-sided.

Mirrored traffic

UserLLMAgentToolDataDecisionHumanGuardrail

What a shadow can and cannot tell you

Read the right-hand column before you plan a shadow, because it is the list of things you will still have to canary afterwards. A shadow is a strong answer to one question and silence on several others.

Question	Can a shadow answer it?	Why
Does the candidate crash on real request shapes?	Yes, strongly	It processes the genuine payload, header and client diversity that synthetic load never reproduces
What does it cost in CPU, memory and connections at real load?	Yes	The workload is real, so the resource profile is real
Does it hold up at production concurrency?	Yes, for reads	It receives the real arrival pattern rather than a generated one (Coordinated Omission: When the Load Generator Lies is the reason generated ones mislead)
Do its responses match the current version?	Partly	Only where responses are deterministic given the same state and normalisable for timestamps and generated identifiers
Is it correct for users?	No	Nobody consumed a response, so no user behaviour, no downstream system and no business metric reacted to it
Are its writes correct?	No — and attempting it is the hazard	A mirrored write is a real write against the same database; the only safe shadow is one that cannot write
Are its external side effects correct?	No	A mirrored request that charges a card charges a real card. These must be blocked, and blocked means untested
Will it behave correctly during a rollout alongside v1?	No	The shadow never coexists as a serving version, so coexistence remains untested (Version Coexistence: N and N+1, in Both Directions)

The ways mirroring becomes the incident

GENERALThese failures follow from duplicating real traffic and are independent of the mirroring implementation — a mesh, a proxy directive and an in-application fan-out produce the same list, though the in-application version also puts the failure inside the process serving users.

A shadow is the only strategy in this module where the testing apparatus itself is the most likely cause of user-visible harm. Every row below is the mirror hurting production rather than the candidate failing.

Trigger	Symptom	Cause	Response
Candidate given ordinary application credentials	Duplicate rows and doubled counters in production data	Mirrored requests execute real write paths against the real database	Read-only database role for the candidate, enforced by the database rather than by the code (Least Privilege in Production)
Egress not blocked	Customers receive duplicate emails; a partner receives duplicate webhooks	Side-effect calls are not part of the response and so are not discarded	Block outbound calls at the network level for the candidate and log the blocks as evidence
Mirroring 100% of traffic	Database connections exhausted; primary path slows	Read load on every shared dependency doubled without a capacity plan	Mirror a fraction, and size the dependency for the fraction you chose (The Connection Budget)
Synchronous mirroring	Real request latency tracks candidate latency	The proxy waits on the copy before completing the primary	Fire-and-forget with its own timeout and bounded in-flight count
Candidate becomes unhealthy under mirrored load	Proxy accumulates pending mirrored requests; memory grows in the proxy	Unbounded mirroring queue with no circuit breaker	Bound the queue and stop mirroring automatically when the candidate fails (Circuit Breaker in Backend terms)
Shadow left running after the decision	Sustained extra cost and extra copies of user data	No end condition was defined when it was started	Give every shadow an expiry and an owner, the same as any temporary production change

How to do it properly

Most important first.

Start by making writes impossible rather than unlikely: give the candidate read-only credentials, or a database role without write permission, so a mistake fails loudly instead of silently persisting (Least Privilege in Production).
Stub or hard-block every external side effect — payment providers, mail, webhooks, third-party APIs — before the first mirrored request, not after the first duplicate email (What Counts as a Secret, and Where It Must Not Be for how the candidate gets different credentials).
Size the dependencies for the extra load, or mirror a fraction of traffic rather than all of it. Doubling read load against a database near its connection limit is its own outage (The Connection Budget).
Make mirroring strictly asynchronous with its own timeout and a circuit breaker, so the shadow cannot affect the primary path even when the candidate is unhealthy.
Diff responses where the endpoint is deterministic, and record the diffs — this is where shadow becomes evidence about behaviour rather than only about stability.
Treat mirrored traffic as production data: it contains the same personal information, and it is now in a second place with second logs (Production Data in Lower Environments).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongOne test

One testEveryone

What contains it

Contained by the discarded response: no user depends on what the candidate returns, so a candidate that fails every request harms nobody. That containment holds only for reads — it is broken completely by any write or external side effect, which escape to everyone, and by the extra load, which reaches every user of the shared dependency.

What can go wrong

Failure modes, including of the mitigation

Shadow writes reaching the production database: duplicate records, incremented counters, consumed idempotency keys, sequences advanced.
Duplicate external effects — the double email is the canonical embarrassing version; the double charge is the expensive one.
Doubled read load pushing a shared dependency into saturation, making the shadow the cause of the incident it was meant to prevent (Connection Pool Saturation: Waiting in Front of an Idle Database in Observability terms).
Synchronous or unbounded mirroring adding the candidate's latency to real requests.
Response diffs that are all noise because responses contain timestamps, generated identifiers or unordered collections — so the diff is abandoned rather than normalised.
A shadow that runs for weeks, silently doubling cost, long after the question it was answering has been answered (Cost Drivers).
False confidence: the shadow validated the read path perfectly, the release broke on the write path, and the team had recorded the shadow as "tested in production".

Misreads this invites

"Shadow testing is risk-free." It is risk-free for users on read paths. It is a live production write path unless you explicitly made it impossible.
"The shadow validated the release." It validated stability and resource behaviour under real traffic. Writes, side effects and user-visible correctness were not validated, because no user saw a response.
"We can shadow everything." Anything that mutates state or has an external effect must be excluded or stubbed, which on a typical service is a large fraction of the interesting behaviour.
"Mirroring is free." It doubles read load on shared dependencies, and it doubles the amount of production data at rest.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

The candidate processed the real traffic mix without crashing, and its resource profile under that mix is known rather than estimated.
Response diffs against the live version are either empty or explained, for the endpoints where diffing is meaningful.
Verified absence of writes and side effects: the candidate's database role has no write permission, and outbound calls to external systems are blocked and logged as blocked.

How you get back

Stop mirroring. There is nothing user-visible to reverse, which is the strategy's defining property.
The exception is anything the shadow actually caused: rows it wrote, effects it emitted, load it added. Those need the same cleanup any bad write needs, and they are harder to find because nobody was expecting the shadow to have done anything (Partial and Logical Data Recovery).

What to automate, and what stays human

Automate the guard rails, not just the mirroring: read-only credentials, blocked egress, bounded concurrency, and an automatic stop when the candidate's error rate or the dependency's saturation crosses a limit.
Automate diff collection and classification, because a diff stream nobody reads is worse than no diff stream — it looks like evidence.
Keep the interpretation of diffs human. Deciding which differences are acceptable is a product question, not a threshold.

What this costs

You pay for a full second copy of the service, plus the extra load on every shared dependency, to obtain evidence about one half of the system.
The setup cost is real: separate credentials, blocked egress, diff normalisation. For most changes it is not worth it; for a rewrite of a high-traffic read path it is the strongest pre-exposure evidence available.
It answers "does it hold up" convincingly and "is it correct" only partially, and the gap between those is where teams over-trust it.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

PLATFORM-SPECIFICTraffic mirroring is a feature of the layer in front of the service: a service mesh, an ingress controller, or a reverse proxy with a mirroring directive. Where none of those exist, teams mirror in application code, which is worse — the application then owns the timeout, the concurrency bound and the guarantee that mirroring cannot delay the real request.
GENERALThe read/write asymmetry is universal. Any duplication of real traffic validates the parts that only compute and invalidates nothing about the parts that persist or emit, regardless of how the mirroring is implemented.