Configuration Drift
Configuration diverges between environments and between instances, and the divergence is invisible until a code path that only exists in one place runs for the first time.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does a service behave differently in production when it is running the same artifact?
Configuration is the input that differs per environment by design, is edited by more people and more systems than code, and is almost never diffed — so divergence accumulates without anyone deciding it should.
Every environment gets values from the same template, so they are consistent apart from the obvious differences like endpoints and sizes.
Values get added in production during incidents and never backported, so a key exists in exactly one place and is exercised nowhere else.
- Values get added in production during incidents and never backported, so a key exists in exactly one place and is exercised nowhere else.
- Values get added in staging during development and never promoted, so a service works in staging and crashes at startup in production (Validate at Startup, Fail Clearly).
- Values are tuned in production under real load — pool sizes, timeouts, concurrency limits — and the tuned values are the ones that matter and the ones nobody wrote down.
- Platform-injected values differ silently: region, zone, instance metadata, service account, injected sidecars, default resource limits.
- Individual instances diverge from each other. A value changed while one instance was restarting, a rollout that half-completed, or a store push that failed for some watchers leaves a fleet where different instances behave differently (Apply Is Not Running).
- Keys accumulate and are never removed, so nobody can tell which of two hundred values are actually read by the current code.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Configuration drift has three axes, and they need different responses. Between environments: staging and production hold different key sets. Between instances: the same environment holds different values on different processes. Between declared and actual: what the repository says and what the process is running.
- The between-instances axis is the most surprising and the least monitored. Any configuration mechanism that applies asynchronously — a store push, a rolling restart, a mounted file update — has a window where the fleet is inconsistent, and a partial failure makes that window permanent.
- Drift is invisible because configuration has no natural diff. Two YAML files with the same keys in different orders and different values look similar to a human, and nothing compares them unless something is built to.
- The damage is delayed. A drifted value causes nothing until a code path reaches it, which may be the first refund, the first failover or the first request from a particular region (Why Local Success Predicts So Little).
Three axes of drift
Naming the axis is most of the diagnosis, because each one presents differently. Between environments looks like "works here, not there". Between instances looks like flakiness. Between declared and actual looks like "I changed it and nothing happened".
| Axis | How it presents | Detector | Typical cause |
|---|---|---|---|
| Between environments | Works in staging, fails at startup in production | Key-set diff per environment | A key added in one place and never promoted |
| Between environments | Works in staging, behaves differently in production | Value diff on the "must match" set (Parity That Is Worth Paying For) | Tuning applied under real load and never backported |
| Between instances | Intermittent failures with no pattern | Configuration version as a metric label per instance | Partial push, half-finished rollout, or a restart during a change |
| Between instances | One zone behaves differently | Per-zone breakdown of the same metric | A zone-scoped override nobody remembers (Regions and Availability Zones) |
| Declared versus actual | A change is made and has no effect | Effective-config dump reporting the source layer | A higher-precedence layer overriding it (Artifact Plus Configuration) |
| Declared versus actual | Reconciliation reverts a value repeatedly | Reconciliation loop logs | Something else writing the same value — an operator, a controller, or a second pipeline |
Making a fleet split visible
The between-instances axis is invisible with normal monitoring because aggregate metrics average across the split. A configuration version carried as a label turns it into something you can see at a glance and alert on.
The rule is simple: after any configuration change, the number of distinct versions in the fleet should return to one within a bounded time. If it does not, some instances never received the change.
1instances by config_version2 3 09:12 cfg-2026-08-19.2 27 <- steady state, one version4 5 09:14 cfg-2026-08-26.1 196 cfg-2026-08-19.2 8 <- expected: push in progress7 8 09:16 cfg-2026-08-26.1 249 cfg-2026-08-19.2 3 <- still converging10 11 09:25 cfg-2026-08-26.1 2412 cfg-2026-08-19.2 3 <- NOT converging: 3 instances13 never got the push14 15 symptom reported: "checkout fails about one time in nine"16 3/27 = 11%The reported symptom and the split match, which is the diagnostic. Without the version label this is an intermittent failure with no pattern, and it typically gets attributed to a flaky dependency for a day or two before anyone checks whether the fleet is uniform (Production Debugging).
Reconciling without breaking what production learned
Correcting configuration drift has the same trap as correcting infrastructure drift: the declared state may be older than the reason production diverged. The difference is often that the divergent value is the correct one.
Production has DB_POOL_MAX=50; the declared configuration says 20. What do you do?
when You have established that 50 was set by mistake or for a situation that has passed.
cost If 50 was tuning that is holding the system up, this is an outage you have scheduled for yourself.
when The production value is correct and was set deliberately under real load — the common case.
cost You are ratifying an unreviewed change; it should still get a reviewer and a reason recorded.
when Nobody can say why production differs, which is the most common state.
cost Time, and the drift persists meanwhile — but applying a value nobody understands is worse.
when The right value is a function of something the environment knows — instance count against a database connection ceiling (The Connection Budget).
cost More machinery, and a derivation that is itself a thing that can be wrong; it removes a whole class of drift in exchange.
How to do it properly
Most important first.
- Derive every environment's values from one declared schema and generate the environment diff mechanically. Key-set differences should be a report, not a discovery (Environment Drift).
- Reject unknown keys in the service's namespace, which turns a leftover from a deleted feature into a visible error instead of a silent passenger.
- Make the running configuration observable: expose a redacted effective dump and a configuration version identifier, and record that identifier as a metric label so a fleet split is visible on a dashboard.
- Backport by default: any value added in production during an incident gets a follow-up to land it in the declared source and in every environment (Manual Production Changes).
- Prune deliberately. Removing a key that nothing reads is a small, safe change, and the accumulation is what makes the surface unreviewable.
- Treat platform-injected values as configuration too. They differ per environment and are usually absent from every diff.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by nothing in the between-environments case — the value is already live everywhere by the time the code path runs. Partly contained in the between-instances case, where only requests landing on the divergent instances are affected, which is also why it presents as intermittent.
What can go wrong
- A diff report that is all noise — every autoscaled count, every rotated version — so the two real differences are never seen (Alert Fatigue).
- Enforcing uniformity on values that must legitimately differ, producing a stream of false positives that trains people to ignore the report.
- Reconciling production toward the declared values and deleting the production tuning that was keeping the system stable.
- A fleet where instances hold different configuration versions and nothing reports it, so an intermittent bug is blamed on flakiness (Flaky Tests).
- Removing an "unused" key that turns out to be read by a rarely-run job, discovered at the end of the month.
- "Same template, so same configuration." The template is the starting point. Everything that happened afterwards — incidents, tuning, platform injection, partial rollouts — is the drift.
- "Drift only matters between environments." The between-instances axis causes the strangest incidents, because the same request succeeds or fails depending on which instance handles it.
- "We store config in Git, so there is no drift." Git holds what was declared. Whether the running process is using it is a separate question, and the answer requires observing the process (State).
- "Extra keys are harmless." An unread key is harmless; a key read by an old code path, or one that shadows a new key with a similar name, is not. And the volume is what makes the whole surface unreviewable.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can produce the key-set difference between any two environments in one command, and the "differs legitimately" set is explicitly declared.
- A dashboard shows the count of instances per configuration version, and it converges to one shortly after every change.
- Startup rejects unknown keys, so leftovers surface as failures rather than accumulating.
- The last configuration-related incident is traceable to a specific unrecorded change with a named owner.
- Reverting a configuration value is easy when it is declared and versioned, and often impossible when it was set in a console — the previous value was overwritten and not recorded (A Config Change Is a Production Change).
- Reconciling drift is itself a change with a blast radius: applying declared values to a production instance that has been tuned by hand can remove the tuning that was holding it up. Read the diff before applying it.
- A fleet split resolves by completing the rollout, not by reverting — reverting during a split can leave a third version in play and make the state harder to reason about.
- Automate the environment key-set diff, the unknown-key rejection, and the per-version instance count. All three are mechanical and all three catch drift before it causes an incident.
- Automate the backport reminder for values changed directly in production.
- Do not automatically overwrite production configuration from the declared source without a human reading the diff. That is the automation most likely to cause the incident it was built to prevent (The Automation Trap).
- Strict uniformity fights the legitimate purpose of configuration, so the "expected to differ" set must be maintained — and maintaining it is ongoing work.
- Rejecting unknown keys breaks the convenient practice of setting a value ahead of the code that reads it, which is a real workflow that now needs a different mechanism.
- A configuration version metric on every instance adds cardinality to your metrics, which has a cost at scale.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALAll three axes exist on any platform. The between-instances axis is worst where configuration is pushed dynamically to running processes, and least bad where configuration is baked into a deployment — there a partial rollout is at least visible as a partial rollout (Rolling: Two Versions, One Database).
- PLATFORM-SPECIFICPlatform-injected configuration differs by runtime and is easy to forget: container platforms inject service account tokens, DNS settings and often sidecar-specific variables, virtual machines inherit image-baked settings, and serverless runtimes provide region and function identity. Each set differs per environment and appears in almost no environment diff.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — asserting environment configuration invariants as tests that fail a build rather than a night.