IaCTOOL-SPECIFICCLOUD-SPECIFICGENERAL
Infrastructure as Code
Describing infrastructure in reviewed, versioned files so it can be reproduced and changed with the same evidence as application code.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
The console already works. Why write the infrastructure down?
The problem
Infrastructure created by hand exists only inside the provider account. Nobody can review it before it happens, reproduce it somewhere else, or answer "what changed" after it breaks.
What teams do first
Build it in the console, where the form fields explain themselves and mistakes are obvious. Write a wiki page describing the steps so the next person can repeat them.
How it breaks
The wiki page is accurate on the day it is written and never again. Nobody notices, because nothing checks it.
How it breaks in production
- The wiki page is accurate on the day it is written and never again. Nobody notices, because nothing checks it.
- The second environment takes weeks instead of hours, and ends up subtly different — a security group rule here, a retention setting there — which is precisely the difference that will produce a production-only bug (Parity That Is Worth Paying For).
- A change nobody reviewed goes out at 14:00 and an outage starts at 14:05. There is no diff to look at, so the incident begins with archaeology in an audit log instead of reading a pull request (Change Correlation).
- Disaster recovery becomes a memory exercise. "Rebuild the VPC" is a sentence, not a procedure, until someone has actually done it (Disaster Recovery as an Operation).
- The person who built it leaves, and the reason a particular flag is set leaves with them.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- You write configuration files that describe the infrastructure you want. The tool reads them, works out what already exists, computes the difference, and calls the provider APIs to close it. Those are the same APIs the console calls — IaC is not a different control plane, it is a different way of driving the same one.
- The value is not automation. Provisioning was already automatable with a shell script. The value is that a desired state expressed in a file can be diffed, reviewed, versioned, and re-applied — three properties a console click has none of.
- To compute the difference the tool needs to know which real resource corresponds to which block of configuration. That mapping is what state is (State), and it is the source of most of the surprising behaviour in this module.
- Nothing about this makes changes safe. It makes them legible and repeatable — which includes repeating a mistake across thirty accounts in ninety seconds (Blast Radius: If This Is Wrong, How Much Does It Affect?).
What the tool is actually doing
TOOL-SPECIFICTerraform and OpenTofu run refresh and diff on your machine or your runner. CloudFormation does the equivalent server-side and you only see the change set; there is no state file you can lose, and correspondingly no state file you can repair.
Strip the vocabulary away and there are four moving parts: your configuration, a record of what the tool last created, the live provider, and a diff between them. Everything confusing about IaC comes from the second one existing at all.
UserLLMAgentToolDataDecisionHumanGuardrail
The three properties, and what each one replaces
It is worth being precise about what is bought, because teams adopt IaC expecting a fourth property — safety — that it does not provide.
| Property | What it replaces | How you check you have it |
|---|---|---|
| Reviewable | Trusting the person clicking | The plan is attached to the pull request and someone other than the author read it |
| Reproducible | A wiki page of steps | The code has been applied into an empty account and produced a working environment |
| Versioned | The provider audit log | You can name the commit that introduced any current setting |
| Safe | Nothing — this is not on the list | Safety comes from the plan gate, provider-side protection and blast-radius separation (Reducing Blast Radius) |
Where the reproducibility claim stops
An apply into an empty account gives you the shape of production and none of its contents. This is not a criticism of IaC; it is the boundary, and teams that do not know where it is plan disaster recovery around a promise nobody made.
Things a fresh apply does not restore
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Region lost, apply code into the recovery region | Infrastructure comes up healthy and empty | Data lives in volumes, buckets and databases, none of which are described by the configuration | Pair every apply-based recovery plan with a tested restore (Restore Drills, RTO and RPO) |
| Rebuild from code six months later | Plan fails or produces different resources | Provider version, module version or upstream image tag moved underneath you | Pin provider and module versions; treat the lock file as part of the artifact (Dependency Pinning) |
| Fresh environment for a new customer | Works, but with different behaviour under load | Provider defaults differ by account age, region or quota, and quotas are not in the code | Encode quotas and account-level settings too, or record them as explicit prerequisites |
| Apply after an emergency console fix | The fix disappears | The code is authoritative and the console change was never brought back into it | Reconcile deliberately, not accidentally (Drift) |
How to do it properly
Most important first.
- Make the code the only path. A single console change that is never brought back into code turns every subsequent plan into a lie (Drift).
- Run the plan in CI on every pull request and post it as a comment, so the reviewer reviews the effect rather than the diff of the source (The Plan: Desired vs Current, Review as a Gate).
- Split state by blast radius: one state per environment, and separate state for the things that are expensive to lose from the things that are cheap to rebuild.
- Give the pipeline the credentials, not the humans. Console access for engineers should be read-only by default, with a break-glass path that is logged (Break-Glass Access, Least Privilege in Production).
- Put provider-side protection on anything holding data — deletion protection, final snapshots, retention locks — so the guard survives the IaC tool being wrong (Destructive Changes: What a Rename Really Does).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongOne region
One testEveryone
What contains it
Separate state per environment and per region, plus provider-side deletion protection. A single global state containing every environment contains nothing at all.
What can go wrong
Failure modes, including of the mitigation
- Partial adoption: half the infrastructure in code, half in the console, and no way to tell which half you are looking at.
- One enormous state where every plan touches every resource, so nobody reads plans any more and the review gate becomes theatre.
- Module abstraction so deep that the reviewer cannot tell what a one-line change will actually apply.
- State lost or corrupted, after which the tool believes nothing exists and offers to create a second copy of production (State).
- The IaC tool itself becomes the bottleneck: a provider upgrade breaks the plan and no infrastructure change can ship until it is fixed.
Misreads this invites
- "IaC means Terraform." Terraform is one implementation. CloudFormation, Pulumi, CDK, Bicep and provider-native templates all express the same idea with materially different state, plan and rename semantics.
- "IaC gives us reproducibility." It gives you reproducible configuration. Data, external state, image contents and provider defaults are all outside it — an apply into a fresh account gives you an empty version of production.
- "Now that it is in code, changes are safe." Code review catches the changes a reviewer can see. The dangerous ones are the changes the reviewer cannot see from the diff, which is exactly why the plan is the artifact under review.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- A plan run against the production branch with no source changes reports no changes. That is the single strongest signal that the code describes reality.
- The provider audit log shows infrastructure mutations coming from the pipeline identity and nothing else (The Audit Trail).
- Someone has actually applied the code into an empty account and got a working environment, recently enough to trust it.
How you get back
- Reverting the commit and applying is not symmetrical with the change you are undoing. Creating a resource then deleting it leaves you without the resource and without its data.
- For additive changes revert-and-apply is a genuine rollback. For anything that replaced or deleted a stateful resource, the rollback path is a restore, not an apply (Partial and Logical Data Recovery).
- Keep the previous plan artifact and the state version. Being able to say exactly what the last apply did is worth more during an incident than the ability to re-run it.
What to automate, and what stays human
- Automate: plan on pull request, apply on merge, drift detection on a schedule, policy checks against the plan (Policy as Code).
- Keep human: approving any plan that destroys or replaces a resource, and deciding whether an emergency console change is worth the drift it creates.
What this costs
- For a single resource you will never change again, the console is faster and the honest answer is to use it — then write it down before it becomes load-bearing.
- You take on the tool as a production dependency: its bugs, its provider versions, its upgrade path, and its state format.
- Abstraction has a cost curve. Modules pay off across many similar environments and get in the way when every environment is different.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICDescribed here in Terraform/OpenTofu terms: local desired state, an explicit state file, and a separate plan step. CloudFormation keeps state server-side as a stack and has no plan file you hold — its change set is server-side too. Pulumi expresses desired state as a program in a general-purpose language, which makes the "diff" a program diff plus a preview.
- CLOUD-SPECIFICWhat is even expressible depends on the provider API. Some resources cannot be updated in place at all, some have no read API and therefore cannot be refreshed accurately, and a few managed services have fields the provider mutates on its own — all of which change what a plan means.
- GENERALThe three properties IaC buys — reviewable diff, reproducible apply, versioned history — hold regardless of tool. Everything below those three is tool detail.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Domains that do not exist yet
- — Testing & Reliability Engineering — what it would mean to test an infrastructure change before applying it, given that the only faithful environment is the one you are changing.