A Successful Deploy Is Not Evidence of a Healthy System
The pipeline reports success when bytes moved and a process answered a health check — which is several layers short of the system doing what it exists to do.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The deploy succeeded. What have I actually learned, and what would I need to check to know the system is healthy?
Deployment tooling reports on itself. It knows whether the artifact was delivered and whether the process started, and it has no opinion about whether the system is doing anything useful. Teams read the green tick as a health verdict because it is the only verdict offered.
The deploy went green and no alerts fired. It worked.
"Green" means the orchestrator reached its desired state: the requested number of processes are running and their health checks return success. Every one of those can be true while the service returns wrong answers.
- "Green" means the orchestrator reached its desired state: the requested number of processes are running and their health checks return success. Every one of those can be true while the service returns wrong answers.
- Health checks are usually shallow by design — a liveness check that touches dependencies causes cascading restarts — so a passing check often proves only that the process can serve a trivial request (Probes: Readiness, Liveness and Startup).
- Alerts are tuned for fleet-wide symptoms. A failure in a subset of endpoints, a slow degradation or a background job that stopped running sits below every threshold you have (Alert on Symptoms, Not on Causes).
- Some failures need elapsed time rather than traffic: a leak, a certificate that expires overnight, a nightly job, a cache that has not turned over yet, a scheduled task that will not run for hours (Cron Jobs in Production).
- Absence of alerts is absence of evidence. The most common silent failure is a deploy that shipped nothing — the pipeline ran, the old artifact is still serving, and everything is green because nothing changed.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Verification is layered, and each layer answers a strictly stronger question than the one below it: the process exists → it answers a health check → it is receiving real traffic → it is answering that traffic correctly → business outcomes still occur → nothing downstream changed → it holds over time.
- The pipeline can answer the first two. Only telemetry and, sometimes, deliberate probing can answer the rest (Using Observability, Not Building It).
- The verification window must outlast the deploy. Failures have characteristic timescales — a crash appears in seconds, a leak in hours, a certificate or a scheduled job in days — and a window shorter than the timescale of the failure you care about cannot see it.
- Verification needs a comparison, not a threshold. "Error rate is 0.3%" is meaningless without the baseline it is being compared against (Canary Analysis: Compared Against What?).
- A specific and common failure deserves naming: the deploy that did not deploy. The pipeline succeeded, the artifact reference was wrong or unchanged, and the running version is still the old one. Everything is healthy because nothing happened (Tags Versus Digests).
Seven layers, each stronger than the last
Most teams stop at layer two and call the deploy verified. The gap between layer two and layer five is where the expensive silent failures live: everything technical is green and the product does not work.
- 11. The intended artifact is running
Confirms the running digest matches the one you meant to deploy.
fails by A mutable tag, an unchanged reference, a failed push — the deploy that deployed nothing.
evidence Digest read from the running system, compared to the release record (Tags Versus Digests).
- 22. The process is alive and ready
Confirms it started, passed startup and reports ready.
fails by A shallow or unconditional health endpoint; readiness passing before dependencies connect (Probes: Readiness, Liveness and Startup).
evidence Readiness true, and the instance actually placed into the load balancer pool.
- 33. It is receiving real traffic
Confirms requests are arriving at the new version, in volume, across the endpoints that matter.
fails by Routing weight applied to a version with no healthy instances, so the step carries nothing.
evidence Request rate on the new version, broken down by endpoint, comparable to the baseline's mix.
- 44. It is answering correctly
Compares error rate by class and latency distribution against a baseline.
fails by Comparison against a threshold rather than a baseline, or against the fleet aggregate (Canary Analysis: Compared Against What?).
evidence Candidate versus baseline, per class, over a defined window — recorded, not glanced at.
- 55. The business outcome still happens
Confirms the thing the system exists to do is still occurring.
fails by No business signal exists, so the entire class of "all green, nothing works" is invisible.
evidence Orders, deliveries, sign-ups or completed jobs at the expected rate — or a synthetic transaction through the critical flow.
- 66. Nothing downstream got worse
Checks the cost the change imposes on dependencies: queries per request, cache hit rate, queue depth.
fails by Service-level signals healthy while database load doubles, which fails later and elsewhere (The Connection Budget).
evidence Downstream load per request compared against the baseline, not in absolute terms.
- 77. It holds over time
Extends observation past the rollout to cover slow failure modes.
fails by The window ended when the rollout did, so leaks, scheduled jobs and expiries are all out of scope.
evidence Stable across at least one full cycle of the slowest relevant process — cache lifetime, job interval, session lifetime.
Layers one and two are the only ones a deployment tool can answer. Everything from three onward requires telemetry, and layer five requires someone to have decided what this service is for.
Green deploy, broken system
Each row is a real class of outcome where every automated signal reported success. They are worth reading as a list of things to add to your own verification, because none of them is exotic.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Deploy referencing an unchanged or mutable tag | The fix appears not to work; behaviour is identical to before | The old artifact is still running; the pipeline succeeded at deploying nothing | Verify the running digest against the intended one as the first verification step (Tags Versus Digests) |
| Health endpoint returns success unconditionally | All instances healthy while every request fails | The check was simplified to stop it flapping and now tests nothing | Readiness should exercise the minimum path that proves the process can serve (Probes: Readiness, Liveness and Startup) |
| Config change with a typo in a rarely used key | Deploy green; one flow broken for a subset of users | Startup validation did not cover the key, and no alert covers that flow | Validate all config at startup and fail fast (Validate at Startup, Fail Clearly) |
| Release stops a background job from being scheduled | Nothing at all, for hours | Absence of work produces no errors, and few systems alert on things not happening | Alert on expected work not occurring, not only on work failing (Cron Jobs in Production) |
| New code doubles queries per request | Service healthy; database saturates an hour later at peak | Verification looked only at the service, not at what it costs downstream | Include downstream load per request in the comparison (The Connection Budget) |
| Slow memory leak introduced | Healthy for hours, then rolling restarts under load | The verification window was shorter than the failure timescale (OOMKilled: Over the Memory Limit) | Extend observation for changes to long-lived state; watch memory trend rather than instantaneous value |
| Feature works; conversion drops to zero | Every technical signal normal | A rendering or validation defect that returns 200 and prevents users from completing the flow | Verify a business outcome, or run a synthetic transaction through the critical path |
The failures that arrive after the deploy is forgotten
The strongest argument for a verification window that outlasts the rollout is that failure timescales are set by the failure, not by the deploy. Reading down this timeline, notice how many hours pass between the change and the symptom — and that by then, "what changed?" is no longer anyone's first question (Change Correlation).
- T+0changeRollout completes. Pipeline green, health checks passing, error rate at baseline. Verification, as commonly practised, is now finished.
- T+2msignalFast failure class: a crash loop or a dependency misconfiguration would have appeared by now. None did.
- T+35msignalCache-driven class: entries written before the deploy begin expiring. New code repopulates them in a new shape; instances that were not restarted start failing (Operating a Cache).
- T+3hsignalSaturation class: memory has been climbing steadily since the deploy. Instantaneous readings were within limits at every point; the trend was the signal (OOMKilled: Over the Memory Limit).
- T+9hsignalScheduled class: the nightly job runs for the first time under the new code and fails. Nothing alerts, because a job not producing output is silence (Cron Jobs in Production).
- T+31hsignalExpiry class: a credential rotated in the release has a shorter lifetime than assumed, and the first refresh fails (Rotation That Applications Survive).
- T+31h05mactionInvestigation begins from the symptom, not from the deploy, because the deploy was yesterday and is no longer a suspect.
- T+31h40mrecoveryDeploy annotation on the dashboard restores the connection between symptom and change, cutting the search enormously (Deploys on the Same Timeline as the Symptom).
Timings are illustrative of characteristic timescales rather than measured. The point is the mismatch: verification windows are set by how long a deploy takes, and failure timescales are set by cache lifetimes, job schedules and credential expiries — none of which know when you deployed.
How to do it properly
Most important first.
- Define, per service, what "verified" means, in layers — and make the top layer a business signal rather than a technical one.
- Verify the version actually running, by digest, from the running system rather than from the pipeline's report.
- Compare against a baseline for errors, latency distribution and the saturation signal that constrains this service (USE: Utilization, Saturation, Errors).
- Watch downstream as well: a deploy that doubles database queries per request is healthy at the service and expensive everywhere else (The Connection Budget).
- Extend the window past the rollout for anything with a slow failure mode, and know which of your failure modes are slow (Production Time Is UTC for the ones tied to the clock).
- Probe deliberately for the paths real traffic does not exercise soon after a deploy — a synthetic transaction through the critical flow is cheap evidence about the part that matters most.
- Annotate deploys on the dashboards operators read, so the correlation between a change and a signal is visible rather than reconstructed (Deploys on the Same Timeline as the Symptom).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Weak verification contains nothing — it is the mechanism that decides how long a bad change runs, so its failure converts a bounded exposure into an unbounded one. What limits it is comparison against a baseline, a business-level signal, and a window that outlasts the deploy.
What can go wrong
- A deploy that shipped nothing: green pipeline, unchanged running version, and hours of confusion when the fix appears not to work.
- A readiness check that returns success before dependencies are connected, so instances take traffic they cannot serve while the rollout reports progress.
- A health endpoint that returns success unconditionally — often after being simplified to stop flapping — making the check a constant rather than a signal.
- Verification that ends when the rollout ends, missing every failure whose timescale is longer than the deploy.
- Business metrics not checked, so a release that quietly stops all sign-ups is green everywhere technical.
- Alert thresholds tuned for total outage, so a partial failure verifies successfully.
- Verification by dashboard glance — no recorded queries, no comparison, no threshold, and therefore no repeatable answer.
- "The pipeline is green, so the deploy is verified." The pipeline verified itself. It has no opinion about whether your system works.
- "No alerts, so it is healthy." Alerts cover the failure modes you anticipated at the severity you configured. New failures are, by construction, not in that set.
- "Health checks passing means healthy." A health check usually proves a process can answer a trivial request. That is a low bar and deliberately so.
- "We verified after the rollout." Verification that ends when the rollout ends is blind to every failure slower than a rollout, which includes most of the expensive ones.
- "The metrics look normal." Compared with what, over what window? Without a baseline and a window, "normal" is a memory (Canary Analysis: Compared Against What?).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The digest running in production matches the digest that was intended, checked against the running system.
- Candidate signals compared against a concurrent or immediately preceding baseline: error rate by class, latency distribution, the constraining saturation signal, and downstream load.
- At least one end-to-end business transaction observed to succeed after the deploy — real or synthetic — through the flow the service exists to serve.
- A verification window that covers the slowest relevant cycle: cache lifetime, scheduled job interval, session lifetime, whichever is longest.
- Verification is what triggers the rollback, so a weak verification produces a late one and a larger blast radius (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- Verify the rollback too, with the same layers. A rollback that reports success and leaves half the fleet on the new version is a common and confusing outcome (Rollback: Only Useful If It Is Actually Safe).
- Keep the pre-deploy comparison available — the baseline you were measured against is also the target state you are trying to return to.
- Automate the layered checks and make them a gate rather than a report. A verification that is only ever read by a human who is already convinced is not a verification.
- Automate the "did it actually deploy" check by comparing the running digest to the intended one; it is trivial and catches an embarrassing class of confusion.
- Keep interpretation of ambiguous business-signal movement human. A dip in orders after a deploy might be the deploy or might be the hour of the day.
- Real verification lengthens deploys, and a long verification window means fewer deploys per day, which pushes batch size up (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Deep health checks give stronger evidence and create cascading-restart risk when a dependency is slow, which is why liveness and readiness are separated in the first place (Probes: Readiness, Liveness and Startup).
- Synthetic transactions are excellent evidence and are also production writes that must be idempotent, cleaned up, and excluded from business metrics (Idempotency in Backends).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe layering — process, health check, traffic, correctness, business outcome, downstream, duration — is independent of platform and of tooling. Only the mechanics of each check differ.
- PLATFORM-SPECIFICWhat "deploy succeeded" means varies: a Kubernetes rollout reports success when the desired replica count is available and ready, a managed platform when its own health probe passes, a script when the last command exited zero. All three are statements about the deployment mechanism and none is a statement about your system — but they fail differently, so knowing which one your green tick is matters.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.