The question this answers
Why does a declarative tool need a state file at all, what exactly is in it, and what happens when two engineers apply at the same time?
The team must be able to change production infrastructure from CI and from two laptops without one run silently undoing another, and without any run being able to read the database password in plaintext just because it needed to add a subnet.
A durable, lockable record of which real resource corresponds to each declared one, so a plan can distinguish "this does not exist" from "this exists and matches" from "this exists and has changed".
Three sources of truth, and the comparison between them
There are three things in play and it is worth being pedantic about them, because almost every state incident is a confusion between two of them. The configuration is what you want. Real infrastructure is what exists. State is what the tool believes it created and what the attributes were last time it looked.
A plan compares all three. Configuration versus state tells the tool what *you* changed. State versus reality — the refresh — tells it what *someone else* changed. The union of those two differences is the plan. This is why deleting the state file is so destructive: without it, the tool has no way to know that the instance in front of it is one it created. Every resource looks new, and the plan proposes to create a second copy of production.
It is also why state cannot be reconstructed by "just reading the account". The provider can list resources, but nothing in the account says *which declared block* a given instance corresponds to. That mapping exists only in state. Rebuilding it means importing every resource by hand, one at a time, matching addresses to ids.
State contains secrets, in plaintext, by construction
This is not a bug and it cannot be configured away. If a resource has an attribute — a generated database password, an access key, a private key, a certificate body — the tool must record its value in order to detect that it changed. sensitive = true suppresses it in CLI output and leaves it fully readable in the state file. Anyone who can read state can read every secret any managed resource ever generated.
The practical consequences are concrete. State never goes in the application repository. The state backend gets encryption at rest, a restrictive read policy that is narrower than the write policy on the infrastructure itself, versioning so a corrupted state can be rolled back, and access logging, because reading state is a credential-access event worth alerting on. Prefer resources that hand you a reference to a secret manager entry over resources that generate the secret as an attribute — see Secrets in Infrastructure.
In an interview, "state is just a cache, it does not really matter" is a red flag with no benefit of the doubt. It says the candidate has never held the lock during an incident and has never thought about who can read the file.
- state:GetObject
- state:PutObject
- lock:Acquire
- lock:Release
- state:GetObject
- state:PutObject
- lock:Acquire
- lock:Release
- state:DeleteObject
- state:DeleteBucketVersioning
Blast radius: Compromise of this role exposes every secret attribute of every managed resource in production — database passwords, generated keys, certificate bodies — and allows a crafted state to make the next apply destroy anything. Read access to state is equivalent to read access to the secret store, and should be granted to the same short list of identities.
Two engineers apply at the same time
Without locking, the sequence is simple and the outcome is bad. Alice reads state, plans, and starts applying. Bob reads the same state a minute later, plans against it, and applies. Alice finishes and writes her state. Bob finishes and writes his, which does not contain Alice's three new resources. Those resources now exist, are billed, and are unmanaged — the tool has forgotten them. The next apply either recreates them and conflicts, or leaves them running forever as orphans nobody can explain.
The fix is a lock held for the duration of the operation, in a store that supports conditional writes: a lock table, a lease on the object, or a state service that provides one. The tool acquires it at the start of plan-and-apply and releases it at the end. A run interrupted mid-apply — laptop closed, CI job cancelled — leaves the lock held, and the recovery is a force-unlock *after* confirming nothing is still running. Force-unlocking a live apply is how you get the two-writer problem back with extra steps.
Remote state with locking is the minimum viable setup for any team above one person. Local state on a laptop is acceptable only for a scratch environment you would be happy to delete. The moment two people can apply, local state is a race condition with a filesystem.
14:02 alice plan (state serial 481) + 3 resources
14:03 bob plan (state serial 481) ~ 1 resource <-- same serial: nobody is holding a lock
14:06 alice apply ok -> writes serial 482 (includes the 3 new resources)
14:07 bob apply ok -> writes serial 482 from HIS copy (does not include them)
14:40 ops plan
Terraform will perform the following actions:
+ cloud_queue.events will be created <-- it already exists. created at 14:06.
+ cloud_queue.events_dlq will be created
+ cloud_subscription.worker will be created
Plan: 3 to add, 0 to change, 0 to destroy.
# the resources are running and billed. state has never heard of them.
# recovery: terraform import, one address at a time, matching real ids by hand.Key points
- State is the map from declared addresses to real resource identities; nothing in the provider account reconstructs it.
- A plan is a three-way comparison: configuration versus state (what you changed) and state versus reality (what someone else changed).
- State contains secret attributes in plaintext.
sensitive = truehides them from output, not from the file. - Two concurrent applies without a lock produce orphaned, billed, unmanaged resources — and the loser's state overwrites the winner's.
- Remote state with locking, encryption, versioning and access logging is the minimum for any team larger than one.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Each resource block has an address (
module.web.cloud_instance.api[0]); state records the address, the provider-assigned id, and the last-known attribute values. - • A run acquires a lock on the state object or in a companion lock table before doing anything that could write.
- • Refresh calls the provider read API per resource to update the recorded attributes, which is where plan time goes.
- • The diff engine compares configuration to refreshed state and emits actions; apply executes them and writes back the new attributes.
- • The state serial number increments on every write; a write against a stale serial is what locking exists to prevent.
- • You own the backend: encryption at rest, versioning, a read policy narrower than the write policy, and access logging.
- • You own lock hygiene, including the judgement call on force-unlock after an interrupted run.
- • You own imports and
state mvwhen resources are created outside the tool or refactored between modules — both are manual, both are risky, both should be done with a plan in hand. - • You own state splitting as it grows, and the cross-configuration output plumbing that replaces the references you lose when you split.
- • You own the recovery drill: restoring a previous state version and confirming what it would do *before* applying it.
- • Lost state: every resource looks new, and the plan proposes to build a parallel copy of production while the original keeps serving traffic.
- • Stale state applied from a laptop, reverting infrastructure changes made from CI over the previous weeks.
- • A held lock from a cancelled CI job, blocking every apply — including the one you need during the incident that cancelled it.
- • A partial apply that fails after creating resources: state is written with what succeeded, and the graph is in a shape neither the configuration nor the previous state describes.
- • State read by someone who needed to debug a plan, and with it every generated password in the environment.
- • Refresh against a resource deleted out-of-band: the tool removes it from state and the plan proposes to recreate it — correct behaviour that looks alarming at 3am. See Drift: When the File and Reality Disagree.
- • Refresh time is the binding constraint: it grows linearly with managed resources, and a plan slow enough to be skipped is a plan that stops protecting anyone.
- • Lock contention is the team-scaling constraint. One state and twenty engineers is a queue; split by change frequency long before that.
- • Cross-state references add coupling in the other direction: a network state that ten application states read is a dependency you cannot refactor casually.
- • Read access to state is functionally read access to every secret the managed resources generated. Scope it like a secret store, not like a build artifact.
- • Write access to state is write access to production, indirectly: a crafted state entry can make the next apply destroy or take over a resource.
- • Version the backend so a corrupted or malicious state write can be rolled back, and alert on state reads by identities outside the expected short list.
- • Prefer resources that reference an external secret manager over resources that generate secrets as attributes, so the plaintext never enters state at all.
- • The backend itself is negligible — an object and a small lock table.
- • The cost that matters is the orphan class: resources that fell out of state stay running and billed with nobody responsible for them, and they are found during a cost review months later.
- • Long refresh times are a productivity cost that compounds: every pull request pays it, and teams respond by planning less often, which is the expensive outcome.
- • State serial history and who wrote each version — this is the audit trail of infrastructure change.
- • Lock acquisition failures and duration; a rising lock wait time is the early signal that the state needs splitting.
- • Access logs on the state backend, treated as credential-access events.
- • A scheduled plan against unchanged main: a non-empty plan is either drift or provider skew, and both want investigating.
- • The signal that lies: a successful apply. It says state was written, not that the resources are healthy or that they are the ones you meant.
- • A managed state service or the provider's own stack service, which owns the backend, the locking and the encryption for you. For most teams this is the better default than running your own bucket.
- • Local state, for a genuinely disposable single-operator scratch environment. It is fine right up until a second person exists.
- • Stateless tooling — a script plus provider-side idempotency (a stack service, or Kubernetes manifests where the cluster holds the state) — when you would rather someone else own the file.
- • For a very small estate, fewer managed resources: not putting the one hand-created DNS zone under management is a legitimate choice that avoids a state entry entirely.
- • Buys the ability to distinguish create from update; costs a durable artifact that is simultaneously critical, secret-bearing and easy to corrupt.
- • Buys safe concurrency through locking; costs a queue, and a force-unlock decision under pressure that can undo the safety you bought.
- • Buys fast, correct planning through cached attributes; costs a refresh whose duration grows until people stop reading plans.
What people believe, and what is true
State is just a cache — it can be regenerated from the account.
The account does not record which declared address a resource corresponds to. That mapping exists only in state, and rebuilding it means importing every resource by hand.
Marking an output sensitive protects the secret.
It suppresses terminal output. The value is stored in state in plaintext, and anyone with read access to the backend has it.
Locking is only needed for large teams.
It is needed the moment two things can apply — which includes one engineer and one CI pipeline, the most common two-writer pair there is.
A failed apply leaves nothing behind.
A partial apply writes state for what succeeded. The environment is now in a shape neither the configuration nor the previous state describes, and the next plan is the only way to find out what.
Go deeper
Overview
State is how the tool remembers which real resource is which declared one. Keep it remote, locked and encrypted.
Practical
Split state by change frequency and blast radius. Network and identity change rarely and hurt a lot; application resources change hourly. Pass values between them as published outputs, never as pasted ids.
Advanced
Rehearse the recoveries before you need them: restore a previous state version and inspect the plan without applying; import a resource created out-of-band; move a resource between modules with state mv. Each of these is a procedure, and the time to learn it is not during the incident.
Internals
State is a JSON document with a serial number, a lineage id and per-resource instances carrying the provider id and the full attribute set. The serial is the concurrency control: a write against a stale serial is exactly what a lock prevents, and lineage is what stops you accidentally applying one environment's state to another.