Image Versus Container
An image is an immutable package; a container is a running instance of it with a throwaway writable layer — which is why nothing you change inside one survives.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the difference between an image and a container, and why does it decide where state is allowed to live?
The two words are used interchangeably in conversation, and the distinction is exactly what determines whether a change persists, whether a fix is reproducible, and whether data survives a restart.
A container is the thing that has my application in it. If I need to fix something, I can exec in, install a package or edit a file, and the container keeps running with the fix.
The change lives in the container's writable layer, which is discarded when the container is removed. A restart, a rescheduling, a node replacement or a scale-up event undoes it, at an unpredictable moment.
- The change lives in the container's writable layer, which is discarded when the container is removed. A restart, a rescheduling, a node replacement or a scale-up event undoes it, at an unpredictable moment.
- It undoes it partially. Only the container you touched has the fix, so a fleet is now heterogeneous in a way nothing records and no dashboard shows.
- Nothing about the running system announces the divergence. The image digest is unchanged, the version endpoint is unchanged, and the behaviour is not (Tags Versus Digests).
- The fix is not reproducible. It exists as something a person did, not as an input to a build, so recreating it requires remembering it (Manual Production Changes).
- Application data written to the container filesystem is in the same position: it looks persistent, and it dies with the container. That failure is silent until the first restart.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- An image is a static, immutable, content-addressed package: an ordered stack of read-only layers plus a config document describing the entrypoint, default environment, working directory, exposed ports and the user to run as. It has a digest and no lifetime.
- A container is a running instance: the image's read-only layers plus a new thin writable layer on top, wrapped in its own namespaces and cgroups, executing a process tree (Process Isolation: One Kernel, Many PID 1s in OS terms).
- The relationship is one-to-many. One image produces any number of containers, each with its own writable layer, its own process and its own lifetime. The image is unaffected by anything any container does.
- Writes go to the writable layer via copy-up: modifying a file that exists in a lower layer copies it upward first, then modifies the copy. The lower layer is untouched, which is why the image stays immutable and why a heavily-writing container accumulates a growing private layer (Copy-on-Write in OS terms).
- When the container is removed, the writable layer is deleted. Persistence therefore requires leaving the container filesystem: a volume, an object store, a database, a queue (Volumes: Storage With a Lifecycle).
- This is the property the whole delivery model rests on. Because the image is immutable and addressable, "what is running" is answerable; because the container is disposable, replacing one is a safe operation rather than a migration.
One image, many containers, one writable layer each
The asymmetry in this picture is the lesson. The layers on the left are shared, immutable and identical everywhere. The thin layer on top of each container is private, temporary and invisible to everything outside that container.
Anything written into one of those top layers exists only for that container's lifetime, which is decided by the scheduler rather than by you.
Which properties belong to which
Most confusion in this area resolves by asking which column a statement belongs in. "It has a digest" is an image statement; "it has an IP address" is a container statement; "I changed it" is almost always a container statement pretending to be an image one.
| Property | Image | Container |
|---|---|---|
| Mutability | Immutable once built | Writable layer on top, discarded on removal |
| Identity | Content digest | Runtime id, valid until it exits |
| Lifetime | Until deleted from the registry | From start until exit or removal |
| Cardinality | One | Many per image, across many hosts |
| Where it lives | Registry, plus a cache on each node | One host, one process tree |
| Network identity | None | Namespace, addresses, ports (Network Namespaces in OS terms) |
| Resource limits | None | Cgroup limits applied at create (Requests and Limits) |
| Where config comes from | Defaults baked in the config document | Environment and mounts supplied at create |
| What survives its death | Everything — it is a stored artifact | Only what was written outside it |
Fixing it inside versus rebuilding
The exec-and-fix is not irrational. It is faster, it is available during an incident, and it appears to work. The reason it is the wrong move is entirely about what happens afterwards, and afterwards can be weeks later.
exec into container-7
edit /app/config/limits.json
restart the process inside
now: container-7 differs from 11 others
digest unchanged, version endpoint unchanged
no record anywhere of what changed
next reschedule silently reverts itcommit the change
-> build -> app@sha256:c71a...
-> deploy the digest
now: every instance identical
the change is in git and in the release record
rollback is the previous digest
the fix survives every rescheduleThe left-hand fix creates a divergence that no signal reports and that reverts itself at a moment nobody chooses, which means the eventual failure has no correlated change to point at. The right-hand path costs a build cycle and produces a fleet where the digest genuinely describes the behaviour — which is the only reason "what is running" is a question with an answer.
How to do it properly
Most important first.
- Treat every container filesystem as ephemeral, including the parts that feel permanent — caches, uploaded files, log files, lock files.
- Put durable state in something designed for it, and be explicit about which of your containers are stateless and which are not (Why Stateful Workloads Are Harder).
- Make changes by building a new image and deploying its digest. If a change is worth keeping, it is worth being in an artifact (Immutable Infrastructure).
- Reserve exec-and-inspect for reading, not writing. Looking inside a container is a legitimate and important operation; changing it is a change with no record (Production Access).
- Write logs to stdout and stderr so the platform collects them, rather than to a file that dies with the container (Using Observability, Not Building It).
- Where a container genuinely must write a lot — a build cache, a scratch directory — mount it explicitly, so the growth is bounded and visible rather than accumulating in the writable layer.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A manual change inside one container affects only the requests that container serves — until it is replaced, at which point the effect vanishes and the behaviour changes again. Nothing contains the confusion that causes; the containment is accidental and the disappearance is the problem.
What can go wrong
- A hotfix applied by exec that disappears at the next reschedule, producing a bug that "came back" with no corresponding deploy (Change Correlation).
- A fleet where some containers have a manual fix and some do not, so behaviour depends on which instance served the request.
- Data written to the container filesystem and lost on restart, discovered when the restart happens rather than when the write did.
- The writable layer growing until the node runs out of disk, taking down every container on that node rather than only the one writing.
- Log files inside the container, which means the logs are gone exactly when the container crashed and you wanted them (Debugging a Container in Production).
- The mitigation failing: a volume mounted at the wrong path, so the application writes to the container filesystem while everyone believes it is writing to durable storage.
- "The container is the deployable." The image is the deployable. The container is what the platform makes from it, and it is disposable by design.
- "Containers are stateless." Containers have a writable layer and can absolutely hold state — badly, invisibly, and until the next restart. Statelessness is a property you design for, not one the runtime provides.
- "Restarting a container gives you a clean state." It gives you the image's state plus any mounted volumes. If the data you wanted gone is in a volume, it is still there.
- "One image, one container." One image can back thousands of containers across a fleet, which is exactly what makes the image the unit worth reasoning about.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Killing any single container and letting it be replaced causes no data loss and no behaviour change.
- The image digest running on every instance of a service is identical, checked rather than assumed.
- A service's durable state can be enumerated: which store, which volume, which external system — with nothing on the list living in a container filesystem.
- Node disk usage does not grow with container uptime, which means writes are going where they were supposed to.
- Rolling back an image is deploying a previous digest; the containers are replaced and nothing needs to be undone inside them (Rollback: Only Useful If It Is Actually Safe).
- A manual change inside a container cannot be rolled back in any meaningful sense — it will be undone eventually, by something unrelated, at a time nobody chose. That unpredictability is the argument against it, more than the change itself.
- Data written into a writable layer and lost is not recoverable. The container is gone and so is the layer.
- Automate the path that makes rebuilding easy, so building a new image is genuinely faster than exec-ing in (Build Performance).
- Automate detection of drift where you can — a read-only root filesystem makes writes fail rather than succeeding invisibly.
- Keep read-only inspection available and easy. Removing an operator's ability to look inside a container in the name of immutability trades a real capability for a symbolic one (Production Debugging).
- Immutability means the smallest fix requires a full build and deploy cycle, which is slower than editing a file — and is the price of knowing what is running everywhere.
- A read-only root filesystem enforces the model and requires every genuinely-needed writable path to be declared, which is work and occasionally surprises a library that writes to a temporary directory.
- Externalising all state simplifies the compute layer and moves complexity into the storage layer, which then needs its own operational attention (Operating a Production Database).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe image/container split is an OCI property and behaves identically across runtimes and orchestrators. What differs is how aggressively containers are replaced: a static single-host deployment may run one container for months, which makes a manual change appear to work, while an autoscaled fleet reschedules constantly and exposes the same mistake within hours.
- PLATFORM-SPECIFICWhere the writable layer lives and how its growth is bounded is a runtime and storage-driver concern. Some platforms enforce a limit per container and some let it consume the node's disk, which turns one badly-behaved container into a node-wide failure. Kubernetes adds ephemeral-storage requests and limits as the mechanism for this; a bare runtime generally has no equivalent unless you configure one (Requests and Limits).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.