The question this answers
What happens when someone changes infrastructure outside the tool, and how do I detect and reconcile it before the next apply makes the decision for me?
The definition of production must stay trustworthy. An engineer reading the repository has to be able to believe it describes what is actually running, or the repository stops being used for anything but ceremony.
A detectable, nameable difference between intended and actual infrastructure, surfaced on a schedule rather than discovered during the next unrelated change.
Drift has two directions and they fail differently
The first direction is reality behind the file. The configuration declares three application servers; there are two, because one was terminated during an incident and nobody put it back. This one is benign in the sense that the tool will fix it: the next plan says + 1 to add and apply restores the third. The danger is capacity — you have been running at two-thirds for six weeks and the autoscaler masked it.
The second direction is reality ahead of the file, and it is the one that bites. During an outage an engineer added a security group rule, or bumped an instance size, or created a fourth server. It worked. Nobody wrote it down. Three weeks later somebody merges an unrelated change to a tag, CI plans, and the plan quietly includes - cloud_instance.api[3] will be destroyed and ~ security rule will be removed. The apply is approved because the pull request was about a tag. The fix that has been holding production together for three weeks is deleted by a change that had nothing to do with it.
That asymmetry is the reason drift detection is not optional in a team that also has an on-call rotation. Convergence is a feature only when what it converges to is current. A file that has fallen three emergency fixes behind reality is a loaded weapon pointed at the next merge.
Detection is a scheduled plan, and nothing else works
Drift detection sounds like it needs a product. It does not: it is a plan, run on a schedule, against the unmodified main branch, in read-only mode, with the result treated as an alert rather than a log line. If nothing has changed in the repository and the plan is not empty, the world has moved. That is the entire mechanism.
What makes it work operationally is the routing. A non-empty scheduled plan should open a ticket or page a channel with the diff attached, because the value is in seeing it *within days* of the change while somebody still remembers making it. A weekly drift report that lands in an inbox nobody reads gives you the same information at the moment it is least useful.
Pair it with the provider audit log. The plan tells you *what* differs; the audit log tells you *who* changed it and when, which is what turns "the security group is wrong" into "an engineer added this at 03:10 during the incident, here is the ticket". See Audit Trails.
$ terraform plan -refresh-only -lock=false # nothing in the repo changed since Tuesday
Note: Objects have changed outside of Terraform
# cloud_instance.api[2] has been deleted
- resource "cloud_instance" "api" {
- id = "i-0f4b19ac"
}
# cloud_security_group.api has changed
~ resource "cloud_security_group" "api" {
+ ingress {
+ cidr = "203.0.113.44/32"
+ port = 8080
}
}
Terraform will perform the following actions:
# cloud_instance.api[2] will be created
+ resource "cloud_instance" "api" { ... }
# cloud_security_group.api will be updated in place
~ resource "cloud_security_group" "api" {
- ingress { cidr = "203.0.113.44/32" ... } <-- this rule is load-bearing. removing it breaks a partner integration.
}
Plan: 1 to add, 1 to change, 0 to destroy.
# audit log, correlated:
# 03:10 role/oncall-breakglass AuthorizeSecurityGroupIngress 203.0.113.44/32
# 03:14 role/oncall-breakglass TerminateInstances i-0f4b19acReconciliation is a decision, not a button
Once drift is visible there are exactly three honest responses, and choosing between them is a judgement about which side is right. Adopt the change: the hand-made edit was correct, so write it into the configuration and the next plan is empty. Revert it: the change was a mistake or a policy violation, so apply and let convergence remove it. Except it: the resource genuinely should not be managed declaratively — a hand-tuned database parameter, a resource another team owns — so remove it from the configuration and record the exception, or use an ignore rule on the specific attribute.
What is not a valid response is applying without deciding, which is the default when drift is invisible. And the fourth non-answer, "lock everyone out of the console so drift cannot happen", fails on contact with an incident: at 03:10 with the checkout flow down, the fastest path is the console, and the correct organisational answer is a break-glass role that is logged and time-limited, not a prohibition people route around.
The cultural piece matters as much as the tooling. Make the reconciliation commit part of incident follow-up, in the same way a post-incident action item is. The team that writes the emergency rule into the repository the next morning never has the three-weeks-later deletion. The team that treats it as paperwork does, roughly once a year, memorably.
| Situation | Response | Mechanism | Cost of getting it wrong |
|---|---|---|---|
| Emergency fix that is genuinely correct | Adopt | Write it into the configuration; next plan is empty | Deleted weeks later by an unrelated merge. |
| Someone bypassed review to widen access | Revert | Apply; convergence removes it. Then ask why the path existed | The exposure persists and now looks sanctioned. |
| Attribute tuned outside the tool on purpose | Except | ignore_changes on that attribute, with a comment saying why | A permanent noisy diff that trains people to ignore plans. |
| Resource owned by another team | Except / unmanage | Remove from configuration; consume it as a data lookup instead | Two configurations fighting over one resource, flipping it on every apply. |
| Nobody knows who made it or why | Investigate first | Audit log lookup before any apply | Deleting something load-bearing to satisfy the file. |
Key points
- Drift has two directions: reality behind the file (the tool fixes it) and reality ahead of the file (the tool destroys it).
- The dangerous case is an undeclared emergency fix removed by an unrelated apply weeks later.
- Detection is a scheduled read-only plan against unchanged main, alerted on, not logged.
- Correlate the plan with the provider audit log: the plan says what, the audit log says who and when.
- Reconciliation is a decision — adopt, revert, or except — and the reconciliation commit belongs in incident follow-up.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Refresh reads every managed resource's current attributes from the provider API.
- • Attributes that differ from those recorded in state are reported as changes made outside the tool.
- • The tool then computes what it would do to make reality match the configuration again — which is the reconciliation proposal.
- • Resources that exist in reality but appear nowhere in state are invisible to this process entirely: unmanaged resources do not drift, they simply are not seen.
- • The provider audit log records the API calls that caused the drift, and is the only source for attribution.
- • You own the schedule, the alert routing and the person who triages a non-empty drift plan.
- • You own break-glass access: a logged, time-limited, alerting role, because prohibiting console access during incidents does not work.
- • You own the reconciliation decision per finding, and the record of exceptions with reasons.
- • You own the ignore rules, and reviewing them periodically — an
ignore_changesadded two years ago is now hiding something. - • You own the discovery problem for unmanaged resources, which drift detection does not cover and cost reports usually find first.
- • A load-bearing emergency rule removed by an apply for an unrelated change; the failure appears in a partner integration, not in your own monitoring.
- • Capacity silently reduced: a terminated instance never replaced, hidden by an autoscaler or by low traffic, until a peak.
- • A permanent noisy diff from an attribute the provider computes differently than the configuration states, which trains the team to skim plans.
- • Two configurations managing the same resource, flipping it back and forth on every apply, each team convinced the other is causing an outage.
- • Unmanaged resources accumulating with no owner — found in a cost review, not by drift detection, because nothing tracks them.
- • Drift volume scales with the number of humans who can touch production, not with the number of resources.
- • Detection cost scales with refresh time, so the same state-splitting that keeps plans readable keeps drift detection frequent enough to be useful.
- • Triage is the human bottleneck: a report with fifty findings gets ignored, so exceptions must be encoded rather than remembered.
- • Drift detection is a security control. An unexpected ingress rule, a widened IAM policy or a disabled log is exactly what it surfaces.
- • Convergence is containment: an accidentally public resource is closed again on the next apply, which limits the window without anyone noticing.
- • Correlate every drift finding with the identity that caused it. A production change made by a human identity outside break-glass is an incident, not a note.
- • A malicious change made *and* written into the configuration will not show as drift — which is why code review on infrastructure paths is a separate, necessary control.
- • Undeclared resources are pure waste: running, billed, and owned by nobody. They are typically discovered in a cost review long after the person who created them left.
- • Drift in the other direction can be a hidden capacity risk with no cost signal at all — running at two-thirds capacity is cheaper, right up to the peak.
- • Scheduled refreshes are free in money and cost plan-time; running them nightly instead of hourly is usually the right trade.
- • A scheduled plan on unchanged main — empty is healthy, non-empty is the alert.
- • Provider audit events filtered to human identities making write calls in production.
- • Break-glass role assumptions: each one should have a corresponding incident and a corresponding reconciliation commit.
- • Count of unmanaged resources per account, from a resource inventory, since drift detection structurally cannot see them.
- • The signal that lies: a green pipeline on the last merge. It proves the last change applied cleanly, not that the world still matches the file today.
- • For a small estate, an inventory query and a human comparison once a month. Not elegant, and genuinely sufficient below a few dozen resources.
- • A continuously reconciling controller (Kubernetes operators, GitOps agents) that converges every few minutes, so drift lasts minutes instead of weeks — at the cost of never being able to make a manual change stick, including during an incident.
- • Provider-side policy enforcement — deny the API call that would create the drift. Prevention beats detection where the rule can be stated precisely.
- • Simply not managing the resources that legitimately change out of band, rather than fighting a permanent diff.
- • Buys a trustworthy repository; costs a triage duty and the discipline of a reconciliation commit after every incident.
- • Continuous reconciliation buys a tiny drift window; costs the ability to hold a manual change, which is exactly what you want at 03:10.
- • Ignore rules buy quiet plans; cost visibility on the attribute you told the tool to stop looking at.
Drift: when reality changed instead
What people believe, and what is true
Drift means someone did something wrong.
Often it means someone fixed production at 3am, correctly. The failure is not the change; it is that the file was never updated to match.
If we forbid console access, drift goes away.
It moves. During an incident people use the fastest path available, and if that is an unlogged shared credential you have lost the audit trail as well as the file.
Drift detection covers everything in the account.
It only compares managed resources. Anything never put in state is invisible to it — and that set only grows.