The Production Loop
Code to build to artifact to release to deploy to observe to incident to learn — and why it is a loop rather than a line.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the full path from a commit to a running system, and where does it close?
Delivery is usually described as a pipeline that ends at deployment, which leaves everything after deployment — the majority of a system's life — without a place in the model.
The pipeline is build, test, deploy. Once it is deployed, the pipeline's job is done and operations takes over.
Deployment is roughly the midpoint, not the end. Most of what determines reliability happens after it.
- Deployment is roughly the midpoint, not the end. Most of what determines reliability happens after it.
- A model that ends at deploy has nowhere to put verification, so "did it work" becomes an informal question answered by silence.
- It has nowhere to put incidents either, so what is learned in an incident does not flow back into how changes are made.
- Rollback becomes an exception rather than a designed step, which is why so many teams discover their rollback path does not work during the incident that needs it.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The loop closes because what you learn in the last three stages changes what you do in the first three. An incident should change a test, a check, a default or an alert — otherwise the loop is open and the same failure recurs.
- Each stage has the same four questions: what goes in, what comes out, what proves it worked, and how you undo it.
- The stages are not equally automatable. Build, test and deploy automate well. Observe partially. Operate, recover and learn are judgement, supported by automation but not replaced by it.
Thirteen stages, four questions each
The value is not the diagram, it is that each stage owes the same four answers. A stage that cannot say what proves it worked is a stage where failures go unnoticed.
- 1Code
A change is written and reviewed.
fails by Change too large to review meaningfully.
evidence Review approved by someone who could have caught the failure mode.
- 2Build
Source becomes an artifact.
fails by Non-reproducible inputs; unpinned dependency.
evidence Same inputs produce the same digest (Reproducible Builds).
- 3Test
Automated checks run against the change.
fails by Flaky tests that train people to re-run (Flaky Tests).
evidence Checks that fail when the behaviour is wrong.
- 4Artifact
An immutable output is stored and addressable.
fails by Mutable tag; rebuilt per environment.
evidence A digest you can point at (Tags Versus Digests).
- 5Release
A decision that this artifact is a candidate.
fails by Conflated with deploy, so there is no decision point (Deployment Is Not Release).
evidence A release record naming artifact, config and migration.
- 6Deploy
The artifact reaches an environment.
fails by All at once, with no coexistence plan.
evidence Instances healthy and serving at the new version.
- 7Run
The system serves traffic.
fails by Readiness lies; traffic arrives before the service can serve (Probes: Readiness, Liveness and Startup).
evidence Requests succeeding at expected rate.
- 8Observe
Signals are collected and compared.
fails by No baseline, so "normal" is unknown.
evidence Candidate compared against baseline, not against zero (Canary Analysis: Compared Against What?).
- 9Operate
Humans run the system day to day.
fails by Toil grows with the service (Toil).
evidence Routine operations are automated and monitored.
- 10Incident
Something goes wrong and is detected.
fails by Detected by customers rather than by alerts.
evidence Time to detect measured, not assumed.
- 11Recover
User impact is stopped.
fails by Rollback path untested (Rollback: Only Useful If It Is Actually Safe).
evidence Impact ended, verified by the same signal that detected it.
- 12Learn
The failure is understood.
fails by Blame instead of contributing factors (Root Cause vs Contributing Factors).
evidence A written timeline reconstructed from evidence.
- 13Improve
The system changes so this class of failure is less likely or less severe.
fails by Action items that say "be careful".
evidence A merged change, not a document.
Where the loop usually breaks
Two joints fail far more often than the rest. The first is deploy → observe: the change ships and nobody checks. The second is learn → improve: the incident is understood and nothing changes.
Both have the same shape. Something completes, so it feels finished, but the step that would have made it useful is the one after it.
How to do it properly
Most important first.
- Walk any change through the whole loop before shipping it, including the last three stages: what would an incident here look like, and what would you do?
- Give every stage an explicit evidence answer. "It deployed" is not evidence that it worked (A Successful Deploy Is Not Evidence of a Healthy System).
- Make sure the learn stage produces a change to the system, not a resolution to be careful (Action Items That Change the System).
- Design the rollback path at the same time as the change, not after it fails (Rollback: Only Useful If It Is Actually Safe).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A missing stage is contained by nothing; it is discovered as a category of failure nobody was watching for.
What can go wrong
- A loop that is really a line: no verification, so failures are found by users.
- A learn stage that produces documents rather than changes.
- Stages that are automated but unobserved — a pipeline that reports success while shipping nothing (A Successful Deploy Is Not Evidence of a Healthy System).
- "The pipeline is CI/CD." CI/CD covers roughly the first half. The operational half is where reliability is actually decided.
- "Verification means the deploy succeeded." A successful deploy means bytes moved. It says nothing about whether the system is healthy.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For the last five incidents, you can point at what changed in the system as a result.
- Every deploy has a defined success signal that is checked, automatically or by a human who is watching.
- The loop itself is the rollback strategy: recover is a stage, not an exception. If your model has no recover stage, your system probably has no rehearsed recovery.
- Automate build, test, package, deploy and the mechanical parts of verification.
- Keep the decision to proceed, roll back or declare an incident with a human, informed by automated signals.
- Formalising every stage is overhead that a small system does not need; the loop scales down by having shorter answers, not by skipping stages.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe stages are stack-independent. How many are automated, and how formally, varies enormously by team size and risk tolerance.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.