Why Local Success Predicts So Little
Everything that makes production hard is absent from the environment where the code was written and reviewed.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
It works locally and in staging. What is production going to do that neither of those did?
Confidence is built in environments that lack the properties which cause production failures, so confidence is systematically higher than it should be.
Test locally, then in staging. If both pass, production will behave the same way — it is the same code.
It is the same code with different traffic, different data volume, different dependency behaviour, different configuration and different concurrency. Every one of those has caused outages on code that passed staging.
- It is the same code with different traffic, different data volume, different dependency behaviour, different configuration and different concurrency. Every one of those has caused outages on code that passed staging.
- Data is the biggest gap. A migration that takes two seconds on 10,000 rows can lock a table for minutes on 200 million (Destructive Migrations).
- Concurrency is the second biggest. Races need concurrent traffic to appear, and staging usually has one user: the person testing.
- Dependencies in lower environments are often stubbed, quieter, or the same shared instance nobody else is hammering.
- Configuration differs by definition — that is what makes it a different environment — and configuration is a common cause of production-only failures (Configuration Drift).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Each pre-production environment is a model of production, and every model omits something. The useful question is not "is it identical" but "which property does this environment preserve, and is that the property I am testing?"
- Staging preserves code paths and integration wiring well. It preserves data volume, concurrency and cost behaviour poorly.
- That is why some classes of confidence can only be built in production, with the blast radius controlled instead of the environment: canaries, flags, shadow traffic (Progressive Delivery: Exposure as a Dial).
What each environment is evidence for
Environments are not a ladder of increasing truth. They are different instruments, each measuring some properties and blind to others. Reading them that way tells you which changes need which evidence.
| Property | Local | CI | Staging | Production |
|---|---|---|---|---|
| Code correctness | Good | Good | Good | Good |
| Integration wiring | Weak | Partial | Good | Good |
| Data volume and shape | None | None | Weak | Only here |
| Concurrency and races | None | Weak | Weak | Only here |
| Dependency failure behaviour | Stubbed | Stubbed | Partial | Only here |
| Cost behaviour | None | None | Weak | Only here |
| Real user behaviour | None | None | None | Only here |
The change classes staging cannot clear
Some changes are systematically under-tested by pre-production environments, and they are disproportionately represented in outage reports. Knowing the list is most of the value.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Schema migration | Writes block; requests time out | Lock duration scales with row count, and staging has few rows | Rehearse against production-scale data; use expand/migrate/contract (Expand, Migrate, Contract) |
| New index on a large table | Elevated latency during creation | Index build competes for I/O at a volume staging never reaches | Build concurrently where the engine supports it; schedule against traffic |
| Change to a hot code path | CPU or allocation rises under real concurrency | Cost per request only matters at production request rates | Canary and compare against baseline (Canary Analysis: Compared Against What?) |
| New dependency call | Tail latency rises; occasional timeouts | The dependency behaves differently under real load than its stub | Timeout and circuit-break before shipping; watch dependency latency on canary |
| Config change only | Immediate failure with no code change | Config differs per environment by definition, so it is the least-tested input | Validate config at startup and treat it as a deployable (A Config Change Is a Production Change) |
| Cache behaviour change | Database load multiplies | Staging hit rates are meaningless — no real key distribution | Roll out progressively and watch downstream load, not just the service |
How to do it properly
Most important first.
- Be explicit about what each environment is evidence for, and what it is not (Parity That Is Worth Paying For).
- Test data-shape-sensitive changes against production-like volume, or accept that you have not tested them.
- For everything staging cannot model, control blast radius in production instead of pretending staging covered it (Reducing Blast Radius).
- Verify in production deliberately, with a defined signal, rather than by waiting for complaints (A Successful Deploy Is Not Evidence of a Healthy System).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
This is the belief that removes containment — it is what makes people ship an unrehearsed change to 100% of traffic at once.
What can go wrong
- Treating a staging pass as a safety guarantee, which is the single most common precondition for a bad release.
- Building an ever-more-faithful staging environment, which costs a great deal and still omits real traffic.
- Copying production data into staging to close the gap, creating a security and privacy exposure (Production Data in Lower Environments).
- "Staging passed, so production is safe." Staging tested the code path. It did not test your data volume, your concurrency or your dependency's bad day.
- "We need staging to be identical to production." Identical is unaffordable and still insufficient. Aim for parity on the characteristics that matter for the change at hand.
- "Production testing is reckless." Uncontrolled production testing is reckless. Canaries and flags are production testing with a bounded blast radius.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For each recent incident, you can say which environment could have caught it and why it did not. If the answer is repeatedly "none", your confidence is coming from the wrong place.
- Changes that are data-shape-sensitive are identified as such before shipping.
- Nothing to roll back here directly, but the lesson shapes rollback: if staging cannot model the failure, the rollback path must be tested in production conditions, not assumed.
- Automate the promotion path so the same artifact moves between environments unchanged (Build Once, Deploy Many).
- Do not automate the judgement of whether staging evidence is sufficient for a given change. That is a per-change decision.
- Higher-fidelity environments cost money and maintenance, and still never reach parity on traffic.
- Verifying in production means real users are briefly exposed to unverified code, which is only acceptable with a small, reversible blast radius.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALApplies wherever pre-production environments exist at all, on any stack.
- SCALE-SPECIFICAt low traffic and small data, staging genuinely is close to production and the gap is narrow. The gap widens with data volume and concurrency — which is precisely when people have most learned to trust staging.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — what a test suite can and cannot establish before production.