Layers and the Build Cache
Why one changed line rebuilds everything below it, why a deleted file is still in the image, and how instruction order decides both.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does changing one line of source sometimes rebuild almost nothing and sometimes rebuild everything?
Build times and image contents are both decided by a mechanism that is invisible in the Dockerfile: each instruction produces a layer, and each layer depends on every layer before it.
A Dockerfile is a script. It runs top to bottom, the result is a filesystem, and reordering the lines is a style preference.
Each instruction produces a filesystem diff that is cached against everything above it. Change an early instruction and every later layer is rebuilt, including the expensive dependency install.
- Each instruction produces a filesystem diff that is cached against everything above it. Change an early instruction and every later layer is rebuilt, including the expensive dependency install.
- The most common ordering — copy the whole source tree, then install dependencies — invalidates the dependency layer on every commit, so the slowest step runs every time even though its inputs almost never change.
- Deleting a file in a later layer does not remove its bytes. The union filesystem records a deletion marker; the earlier layer still contains the file and anyone with the image can read it (Secrets in CI).
- A layer that fetches from the network with no pin captures whatever the network returned at build time, and the cache then serves that content indefinitely to builds that would otherwise have got something newer (Dependency Pinning).
- Cache availability differs between a laptop and CI. A build that is fast locally because everything is cached can be a cold build on every CI runner, which is a different pipeline than the one anyone measured (Caching in CI).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- An image is an ordered stack of read-only layers plus a config document. Each layer is a diff — files added, changed or marked deleted relative to the layers below — stored as its own content-addressed blob and shared between every image that contains it (Artifact Registries).
- At run time the layers are stacked by a union filesystem and a thin writable layer is placed on top. Reads fall through the stack to the topmost layer that has the file; writes copy the file up into the writable layer first (Copy-on-Write in OS terms).
- The build cache keys each instruction on the identity of the parent layer plus the instruction itself. For a copy instruction it also includes a checksum of the files being copied; for a run instruction it does not look inside the command at all — the same command string is a cache hit even if the world it queries has changed.
- Because the key includes the parent, cache invalidation is a suffix operation: the first instruction whose key changes invalidates itself and everything after it. Order therefore determines cost.
- Deletion is recorded as a whiteout entry in the upper layer. The lower layer is untouched and remains in the image, which is why a secret written and then removed in a later instruction is still extractable from the artifact.
Order decides what gets rebuilt
Both files below produce the same running application. They differ only in where the source copy sits relative to the dependency install, and that decides whether the dependency install runs on every commit.
The cache is a prefix match: the build reuses layers until the first instruction whose key differs, then rebuilds everything after it. Putting the most volatile input first therefore guarantees a full rebuild every time.
FROM node:22-slim WORKDIR /app COPY . . <- changes on every commit RUN npm ci <- therefore reruns on every commit RUN npm run build CMD ["node", "dist/server.js"]
FROM node:22-slim WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci <- reruns only when the lockfile changes COPY . . RUN npm run build CMD ["node", "dist/server.js"]
The cache key of every instruction includes the layer above it, so the first changed instruction invalidates the whole suffix. In the left version the source copy sits above the dependency install, so a one-character change to a comment reinstalls every dependency. In the right version the install layer's inputs are the lockfile and nothing else, so it is reused until the dependencies genuinely change.
What invalidates a layer, and what does not
The surprising entries are the last two. A run instruction is cached on its command string, so a command that reaches the network returns whatever it returned the first time, for as long as the cache survives — which is a correctness problem disguised as a performance feature.
| Change | Invalidates? | Why it matters |
|---|---|---|
| Any earlier instruction changed | Yes, and everything after it | Cache is a prefix match; order is cost |
File contents in a COPY source | Yes | Copy instructions include a checksum of what they copy |
File mode or path in a COPY source | Yes | Metadata is part of the diff |
| A file excluded by the ignore file | No | Which is why an ignore file is a caching tool as well as a size tool |
| Base image tag repointed | Only on a fresh resolve | A cached parent layer keeps the old base until something forces re-resolution (Tags Versus Digests) |
| A build argument used by the instruction | Yes | Which is how per-environment builds sneak in (Build Once, Deploy Many) |
| The upstream package index changed | No | The RUN string is unchanged, so the layer is reused and ships stale packages |
| Time passing | No | There is no expiry; a cached layer is valid until something invalidates its key |
A deleted file is still in the image
This is the layer property with a security consequence rather than a performance one, and it surprises people who read a Dockerfile as a script.
The rule to carry: anything that exists in any layer exists in the image. The only safe pattern is never writing the sensitive value into a layer at all — either mount it for the duration of one instruction, or supply it at run time.
1# 1. Still in the image. The RUN below writes a whiteout marker2# in a new layer; the token remains readable in the layer above.3COPY .npmrc /root/.npmrc4RUN npm ci5RUN rm /root/.npmrc6 7# 2. Still in the image, and now also in the metadata.8ARG NPM_TOKEN9RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > /root/.npmrc && npm ci10 11# 3. Never becomes a layer: mounted for this instruction only.12# syntax=docker/dockerfile:113RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ciThe second form is the one that looks safest and is the worst: build arguments are recorded in the image config, so the value is available to anyone who can inspect the image, without even extracting a layer. If a credential has ever been in a published layer, the response is rotation, not deletion (Rotation That Applications Survive).
How to do it properly
Most important first.
- Order instructions from least to most frequently changing: base, system packages, dependency manifests, dependency install, application source, build.
- Copy dependency manifests alone, install, and only then copy the source. That single split is usually the largest build-time improvement available (Build Performance).
- Never write a secret into a layer, even temporarily. Use the builder's secret mechanism, which mounts the value for one instruction without persisting it, or pass it in at run time (Secrets in CI).
- Pin what layers fetch. An unpinned package install makes the layer non-reproducible and makes the cache a source of silent staleness (Reproducible Builds).
- Combine instructions when the intermediate state is worthless, and separate them when the intermediate state is a useful cache boundary. Those pull in opposite directions and the boundary should be where change frequency changes.
- Share a base image across services so the common layers are already present on every node, which makes cold pulls cheaper without shrinking anything (What Image Size Actually Costs).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A stale layer ships whatever it contains to every instance, contained only by the rollout strategy. A leaked secret in a layer is contained by nothing at all — the image is distributed by design (When Secrets Fail).
What can go wrong
- A stale cached layer serving an old dependency set, so a build "succeeds" and produces yesterday's software with today's commit.
- A secret in a layer, discovered by an image scan or by someone extracting it, long after the tag was overwritten.
- A cache key that includes a timestamp or a build id, defeating the cache entirely and making every build cold.
- Cache shared between untrusted builds, which lets one build poison a layer another build consumes (CI Security).
- Aggressive layer combining that removes every useful cache boundary, so every change rebuilds everything.
- The mitigation failing: careful ordering in the Dockerfile combined with a CI runner that has no persistent cache, so the ordering buys nothing and everyone believes builds are fast.
- "Fewer layers means a smaller image." It means fewer manifest entries. The bytes are the same unless combining lets you delete something *within* a single instruction, before the layer is written.
- "I deleted the file in the next line, so it is not in the image." It is. The deletion is a marker in a later layer; the bytes remain in the earlier one.
- "The build cache makes builds reproducible." It makes them fast. Reproducibility comes from pinning inputs, and the cache can actively hide the fact that inputs are not pinned.
- "Layer sharing means image size does not matter." It means the *shared* layers are pulled once per node. The layers unique to your image are pulled every time they change (What Image Size Actually Costs).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A source-only change rebuilds only the layers after the source copy, visible as cache hits in the build log.
- Extracting each layer of a published image finds no credential, key or token in any of them.
- Two builds of the same commit produce the same layer digests, at least for the deterministic layers (Reproducible Builds).
- Build duration in CI is close to build duration locally for the same change, which means the cache is actually shared.
- Layers are immutable, so there is nothing to roll back within an image — you deploy a previous digest instead (Tags Versus Digests).
- A leaked secret in a published layer cannot be rolled back at all. Deleting the tag does not delete the blob, and anyone who pulled it has it. The only real response is rotation (Rotation That Applications Survive).
- Automate cache configuration in CI — a shared, versioned cache with an explicit scope — so the ordering work in the Dockerfile actually pays.
- Automate a scan of published images for credentials and for the layer that introduced them.
- Do not automate a periodic cache wipe as a fix for staleness. That trades a correctness problem for a cost problem and leaves the unpinned dependency in place.
- More layers means finer-grained caching and more metadata per image; fewer layers means smaller manifests and coarser rebuilds. The right split follows change frequency rather than a target count.
- A shared build cache makes builds fast and becomes a trust boundary between everything that writes to it.
- Optimal ordering can conflict with readability — the natural narrative order of a Dockerfile is often the worst caching order.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICThe layer model is an OCI image property and is the same everywhere. What differs is the builder: BuildKit adds parallel stage execution, cache mounts that persist a directory across builds without putting it in a layer, and a secret mount that never becomes a layer — none of which exist in the classic sequential builder. Whether any of it works in CI depends on whether the runner has a persistent or remote cache, which is usually the deciding factor and is not a property of the Dockerfile.
- GENERALThe union-filesystem behaviour — reads fall through, writes copy up, deletions are markers — is shared by the common storage drivers. Their performance characteristics differ; the visible semantics do not.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.