Drift
Reality diverging from what the code says — how it happens, which of it is legitimate, and why the next apply is the dangerous moment.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The code says one thing and production does another. Which one is wrong?
Infrastructure can be changed by things that are not your IaC tool: a human in a console during an incident, another automation, an autoscaler, or the provider itself. Once that happens, the code is a description of the past.
Run an apply. Whatever the code says is authoritative, so applying makes reality match it again.
The drift was an emergency fix. Applying reverts it and re-opens the incident, this time with less patience in the room.
- The drift was an emergency fix. Applying reverts it and re-opens the incident, this time with less patience in the room.
- The drift is a field another system legitimately owns — a scaled-out task count, a tag applied by a policy engine. Applying fights that system in a loop, one change per side, forever.
- The drift is a deletion. Applying recreates the resource empty, which for anything holding data is a fresh disaster on top of the original one.
- Nobody knew there was drift until an unrelated change was applied and dragged six months of accumulated divergence with it.
- Drift accumulates silently in the environment nobody applies to often, which is usually the one that matters most.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Drift is detected during refresh: the tool reads live attributes and compares them to the cached values in state. A difference in either direction is drift.
- Direction matters and the plan does not label it. "Configuration says A, reality says B" reads the same whether someone changed reality or someone changed the configuration and never applied it.
- Some drift is structural rather than accidental. Providers set defaults, normalise values, and mutate fields on their own schedule; a plan that shows a permanent one-line diff on every run is usually this, not a person.
- The mechanism that removes drift — apply — is also the mechanism that destroys the information about what the drift was. Detect and record before you reconcile.
- A tool that does not refresh cannot see drift at all. Skipping refresh for speed makes plans faster and blinder.
Four kinds of drift that look identical in a plan
A plan shows you a difference. It does not show you which side moved or why, and the response is completely different in each case.
| Kind | What happened | Right response | Wrong response |
|---|---|---|---|
| Emergency change | A human fixed production during an incident | Codify it in a pull request; keep the fix | Apply, reverting the fix and restarting the incident |
| Owned elsewhere | An autoscaler or policy engine owns that field | Ignore the field explicitly in code | Apply repeatedly, fighting the other system |
| Provider normalisation | The provider rewrote or reordered a value on read | Match the provider's form, or ignore the field | Treat the permanent diff as background noise for everything |
| Abandoned change | Someone edited code and never applied it | Apply it, or delete it — but decide | Leave it, so the next unrelated apply carries it |
Ignoring a field is a decision, not a workaround
When another system genuinely owns a field, saying so in code is better than any amount of process. It removes the noise, records the ownership, and stops the two systems fighting.
1resource "aws_ecs_service" "api" {2 name = "api"3 cluster = aws_ecs_cluster.main.id4 task_definition = aws_ecs_task_definition.api.arn5 desired_count = 36 7 lifecycle {8 # The autoscaler owns this at runtime. Without this, every apply9 # resets capacity to 3 -- including mid-incident, at peak.10 ignore_changes = [desired_count]11 }12}The value in the code still matters: it is the count used when the resource is first created. The ignore only stops subsequent applies from resetting it.
How an emergency fix becomes a second outage
This is the single most common drift incident, and it has nothing to do with carelessness. Every step is reasonable in isolation.
- 02:14signalAlert: API error rate climbing; a security group rule is blocking traffic from a new subnet
- 02:19actionOn-call opens the console via break-glass and adds the missing rule by hand
- 02:21recoveryError rate recovers. Incident mitigated
- 02:40actionIncident closed. The console change is noted in the channel and not in a pull request
- 11:30changeUnrelated change merged: a new tag on a different resource in the same state
- 11:31actionPlan shows two changes. The reviewer reads the tag and approves
- 11:33changeApply removes the hand-added rule, because the code never had it
- 11:35signalSame alert, same symptom, no obvious deploy to blame — the change that caused it was a tag
The failure is not the console change; that was the right call at 02:19. The failure is that nothing forced the change back into code before an unrelated apply carried it away. Scheduled drift detection would have caught it at 03:00, hours before anyone was near the keyboard.
How to do it properly
Most important first.
- Detect on a schedule, not only when someone happens to make a change. A nightly plan against every environment, reporting non-empty results, converts drift from a surprise into a queue (Toil is the risk here — keep the report small enough to act on).
- Classify before reconciling. Emergency human change, another system's field, provider normalisation and abandoned configuration are four different situations with four different responses.
- For fields another system owns, tell the tool to ignore them explicitly. Ignoring is a decision recorded in the code, which is much better than an apply loop nobody understands.
- For emergency changes, the fix is a pull request that codifies what was done, not an apply that undoes it. That is the whole point of break-glass being logged (Break-Glass Access).
- Make the console read-only for routine work, so drift becomes an event rather than a background process (Manual Production Changes).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Scheduled detection, a read-only console, and a destroy gate on the reconciling apply. Undetected drift in a shared state is contained only by the next person who reads a plan carefully.
What can go wrong
- Drift detection that reports on every run because of provider normalisation noise, and is therefore ignored — the same failure as a noisy alert (Alert Fatigue).
- Auto-remediation that applies on detection, which reverts emergency fixes automatically and at the worst possible time.
- An
ignore_changeslist that grew until the code no longer describes anything meaningful. - Drift in a resource that was deleted outside the tool, where reconciliation means recreation and recreation means an empty database.
- Detection running with reduced permissions, so it reports "no drift" for resources it cannot read.
- "All drift is bad." Fields owned by an autoscaler or a policy engine drift by design. What is bad is drift you did not know about (Drift in Cloud & Infrastructure looks at the same phenomenon from the provisioning side).
- "Drift means someone did something wrong." Sometimes it means someone fixed an outage at 03:00 with the only tool available. Treat the drift as information about a gap in the automation.
- "Continuous reconciliation solves drift." It converts drift into a fight between two systems. That is better only when you are certain your side should always win — which is exactly the assumption that reverts the emergency fix.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A scheduled plan across environments reports empty, and you can tell the difference between empty and errored.
- Every non-empty drift report has a resolution recorded: codified, ignored deliberately, or reverted with a named approver.
- The audit log shows console mutations only from break-glass sessions with tickets attached (The Audit Trail).
- Reconciling drift is itself a change and needs the same gate as any other apply. If the plan contains a destroy, it goes through a human.
- If reconciliation reverted something it should not have, the rollback is to reapply the emergency change — by hand if necessary — and then codify it properly once the incident is over.
- Automate: detection, classification into "noise" and "real", and reporting into a queue that a person owns.
- Keep human: deciding what to do about each real instance. Automatic reconciliation is one of the clearest cases in this domain of automating a judgement you should not (The Automation Trap).
- Frequent detection costs provider API calls and rate limit budget, and on large estates that is a real number.
- Locking the console down slows genuine emergencies. The answer is a fast, logged break-glass path, not permanent write access.
- Ignoring fields keeps the plan clean and hides real changes in those fields. Every ignore is a small blind spot you accepted.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICTerraform/OpenTofu detect drift only when something runs a refresh, so drift is invisible between runs. A Kubernetes controller reconciles continuously and removes drift within seconds without asking — different guarantee, and it means an emergency
kubectl editon a managed object is reverted before you finish reading the output (Reconciliation: The Loop Under Everything). - CLOUD-SPECIFICProvider-normalised fields differ by provider and by resource: policy documents reordered on read, tags injected by an account-level rule, capacity fields owned by a managed autoscaler. Which fields are noise is something you learn per provider, not in general.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — detection cadence as a reliability property: how long a divergence can exist before something notices.