The Container Lifecycle
The full path from a build context to a process serving traffic, and back to a stopped container — with the failure that belongs to each hop.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the complete sequence between docker build and a process serving requests, and which step is failing when a deploy does not work?
Container failures are reported as one undifferentiated symptom — "the container will not start" — which covers at least six distinct steps with entirely different causes and fixes.
It is two commands. Build the image, run the image. If the container is not running, read the application logs; that is where the problem will be.
Several steps happen before the application produces a single log line: context upload, layer build, push, pull, create, start. A failure in any of them leaves nothing in the application log because the application never ran.
- Several steps happen before the application produces a single log line: context upload, layer build, push, pull, create, start. A failure in any of them leaves nothing in the application log because the application never ran.
- Create and start are separate operations. A container can exist with a filesystem, an assigned name and a network attachment, and never have executed its entrypoint.
- The most common production start failure is not in the code — it is a missing environment variable, an unmountable volume, an unreadable secret or a wrong user id, all of which happen at start and produce a short, unhelpful message.
- Stop is not one event either. The runtime sends a termination signal, waits a configured grace period, then kills. An application that ignores the first sees only the second, which it cannot handle (Graceful Shutdown).
- Removal discards the writable layer. Anything the application wrote to its own filesystem — including logs written to a file — is gone, and it is gone precisely when you wanted to read it (Debugging a Container in Production).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The lifecycle has two halves that are usually conflated. The build half turns a context and a Dockerfile into layers, a manifest and a digest in a registry. The run half turns that digest into a process on a specific host.
- On the run half, the runtime pulls missing layers, assembles a root filesystem by stacking the read-only layers with a new writable layer on top, creates namespaces and cgroups for the container, and then execs the entrypoint as PID 1 in that namespace (Containers Are Processes With the Kernel’s View Narrowed in OS terms).
- A container is not a service. It is one process tree, with a lifetime, that exits. Everything that makes it look long-lived — restarts, replacement, scheduling — is the layer above it (The Container Lifecycle is about the container; Pods: The Unit That Gets Scheduled and Deployments: Declaring What Should Be Running are about keeping one running).
- The state machine is small: created, running, paused, exited (with a code), removed. Almost all production confusion comes from not knowing which of those a workload is in, and from exited containers being removed before anyone read the code.
- Restart policies operate on the exit. A container that exits non-zero and is restarted repeatedly is a crash loop, and the interesting information is in the *previous* container, not the current one.
Every step, and how each one fails
The value of the sequence is diagnostic. Naming the step narrows the cause more than any amount of reading application logs, because most of these steps produce no application logs at all.
- 1Context
The build client sends the files the build may read.
fails by A file present locally and excluded in CI, or a huge context that makes every build slow.
evidence The build fails on a missing file rather than silently using a stale one.
- 2Build
Each instruction produces a layer; unchanged prefixes come from cache.
fails by Cache invalidated too early, or an instruction that reads the network unpinned (Layers and the Build Cache).
evidence A digest, and a cache hit ratio that matches what changed.
- 3Push
Layers and manifest are uploaded; the registry returns the digest.
fails by Auth expiry or a partial upload reported after the build step already passed.
evidence The digest read back from the registry matches the one pushed.
- 4Pull
The node fetches layers it does not already have.
fails by Credentials, rate limits, network path, or a deleted manifest (Artifact Retention).
evidence The node reports the image digest it resolved.
- 5Create
Root filesystem assembled, namespaces and cgroups created, mounts attached.
fails by Unavailable volume, missing secret, invalid user or resource request that cannot be satisfied.
evidence A created container with the mounts it expects.
- 6Start
The entrypoint is executed as PID 1 in the container's namespaces.
fails by Entrypoint not executable, wrong architecture, missing shared library, config absent at start-up.
evidence The application produces its first log line.
- 7Ready
The process reports it can serve, and traffic is routed to it.
fails by Readiness passing before dependencies are usable, so traffic arrives too early (Probes: Readiness, Liveness and Startup).
evidence Requests succeeding, not merely a process existing.
- 8Stop
Termination signal, grace period, then kill.
fails by Signal ignored or swallowed by a wrapper, so the grace period elapses and work is dropped (PID 1 and Signals).
evidence Shutdown logged, in-flight requests completed, exit code 0.
- 9Remove
The writable layer is discarded.
fails by Evidence removed before it was read.
evidence Nothing was needed from inside, because logs and metrics left the container while it ran.
The two halves, and where the identity crosses
The registry is the seam. Everything left of it is a build concern that can be retried freely; everything right of it happens on hosts serving users, where a retry is a production event.
The digest is what crosses. A pipeline that carries a tag across this seam instead has reintroduced ambiguity at the exact point where the two halves stop being able to check each other (Tags Versus Digests).
Which step is it?
Every row here is reported the same way by the person who noticed it. The middle columns are what actually distinguishes them, and all of it is available in seconds.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Image not found | Never starts; runtime reports pull failure | Wrong reference, deleted manifest, or a registry the node cannot reach | Check the reference and the node's registry path, not the application (Artifact Registries) |
| Auth failure on pull | Some nodes start, others do not | Credential expired or not present on that node | Use workload identity rather than a stored token (Workload Identity) |
| Exec format error | Starts and exits instantly | Image architecture does not match the node | Build a multi-architecture image and pin the index digest |
| Missing shared library | Exits instantly with a loader error | Binary built against a libc the runtime base does not have | Match build and runtime bases, or link statically (Multi-Stage Builds) |
| Missing configuration | Exits on start with a short message, restarts, repeats | Required environment variable or secret absent in this environment | Validate at start-up and name the missing key (Validate at Startup, Fail Clearly) |
| Volume unavailable | Created, never started | Mount cannot be attached on this node | A storage or scheduling problem, not an application one (Volumes: Storage With a Lifecycle) |
| Ready too early | Starts, receives traffic, returns errors briefly | Readiness does not test the dependencies it needs | Make readiness reflect ability to serve (Probes: Readiness, Liveness and Startup) |
| Killed after grace period | Errors during every rollout, none between them | Termination signal not handled or not forwarded | Fix the process model, then the drain (Graceful Shutdown) |
| Killed for memory | Exit code 137, no application error | Cgroup memory limit exceeded | Distinguish an OOM kill from a grace-period kill before tuning anything (OOMKilled: Over the Memory Limit) |
How to do it properly
Most important first.
- Learn to name the step. "Pull failed", "created but not started", "started and exited 1", "started and never became ready" are four different incidents with four different first questions.
- Read the runtime's events before the application's logs. Events cover the steps that happen before the application exists (Reading a Broken Workload for the orchestrated case).
- Log to stdout and stderr so output survives the container. A log file inside the writable layer disappears with the container that produced it.
- Make start-up fail loudly and immediately on missing configuration, rather than starting and failing later on first use (Validate at Startup, Fail Clearly).
- Set the stop grace period deliberately, and know what your application does with the termination signal before you tune it (PID 1 and Signals).
- Keep the exit code. It is the cheapest single piece of diagnostic information in the whole lifecycle.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rolling deployment contains a bad image to the instances replaced so far, and health checks stop the rollout if the new containers never become ready. Nothing contains it if the failure is at stop rather than start — every replaced instance drops its in-flight work (Rolling: Two Versions, One Database).
What can go wrong
- Build succeeds locally and fails in CI because the build context differs — an ignored file, an untracked file, a cached layer.
- Push succeeds and the pipeline reports the build step's success, so a failed push is discovered at deploy time (Artifact Registries).
- Pull fails on a subset of nodes: expired credentials, a rate limit, or a node in a different network path.
- Container created and never started because a mount is unavailable, which presents as a container that "exists" and does nothing.
- Started and exited immediately, with the reason in a message the orchestrator overwrote on the next restart.
- Stopped by SIGKILL after the grace period, so no drain happened and in-flight requests were dropped (Draining: Stopping Without Dropping).
- The mitigation failing: an aggressive restart policy that hides a crash loop as "the service is up", because the process is running for a few seconds at a time.
- "The container is running, so the service is up." Running means the entrypoint process exists. Whether it can serve is a separate question with a separate signal (Probes: Readiness, Liveness and Startup).
- "Restarting fixed it." Restarting reset the state that was wrong. The state will become wrong again, and now you have less information about how.
- "Containers are lightweight VMs." They are processes with a different view of the system. That difference is the whole subject of Image Versus Container and of the OS domain's VM vs Container: Where the Boundary Is.
- "The logs are in the container." They are in the container only if you put them there, which is the mistake. They should be in the platform.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For any unhealthy workload you can state its lifecycle step and its exit code without guessing.
- The digest that was pushed is the digest that was pulled, and both match the release record (Tags Versus Digests).
- Application output is retrievable after the container is gone, because it went to the platform's log pipeline rather than to a file inside the container.
- A deliberate stop shows the application handling the termination signal in its logs, followed by a zero exit code.
- Reversing a container change is deploying the previous image digest, which is fast because the layers are usually still present on the nodes (Rollback: Only Useful If It Is Actually Safe).
- What does not roll back is anything the container wrote outside itself: a volume, a database, a queue. The container lifecycle is reversible; its side effects are not (Why Migrations Are the Dangerous Change).
- Automate the build-push-verify sequence as one unit so an artifact never exists without having been pushed and read back.
- Automate start-up validation and health reporting so the platform can distinguish "started" from "able to serve" (Probes: Readiness, Liveness and Startup).
- Do not automate away the exit code. Whatever the platform does on failure, the code and the previous container's output must remain retrievable, or every crash loop becomes an archaeology exercise.
- The lifecycle gives strong reproducibility — the same digest produces the same filesystem everywhere — at the cost of an entirely ephemeral local filesystem, which changes how logging, caching and temporary files must be designed.
- Restart policies improve availability and hide failure. The same mechanism that keeps a flaky service up also delays the moment anyone notices it is flaky.
- A short grace period makes deploys fast and makes drains impossible; a long one makes drains reliable and makes rollbacks slow (Graceful Shutdown).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICThe steps are the same across container runtimes; the surfaces differ. Docker, containerd and CRI-O expose the same create/start/stop sequence with different CLIs and different event streams, and podman runs it without a long-lived daemon. On an orchestrated platform the runtime's lifecycle is wrapped by another one — pod phases on Kubernetes, task states on a managed container service — which adds scheduling, placement and readiness states that the runtime itself knows nothing about.
- GENERALThe two-half structure — a build half producing an addressable artifact and a run half instantiating it on a host — holds for every container platform, and the failure families map one-to-one onto the steps regardless of tooling.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.