Multi-Stage Builds
Compile in one image, ship another — so the toolchain, the source and the build credentials never reach production.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do you build inside a container image without shipping the compiler, the source tree and everything the build needed?
A build needs a toolchain, source, package managers and often credentials. A runtime needs none of those. In a single-stage build they are the same filesystem, so everything the build touched ships to production.
Install the toolchain, copy the source, build, then delete the toolchain and the source in a final cleanup step. The image ends up with only what is needed.
Deleting in a later layer does not remove the bytes. The compiler, the source and anything else are still in the earlier layers and still extractable from the published image (Layers and the Build Cache).
- Deleting in a later layer does not remove the bytes. The compiler, the source and anything else are still in the earlier layers and still extractable from the published image (Layers and the Build Cache).
- Any credential used during the build — a package registry token, an SSH key for a private dependency — is in a layer for the same reason, and deleting it changes nothing (Secrets in CI).
- The runtime image inherits the entire attack surface of the build image: package managers, compilers, network tools and a shell, all available to anything that achieves code execution (The Delivery Chain as Attack Surface).
- Cleanup steps are fragile. They are written once, they miss things, and they silently stop matching the build as it evolves.
- The image ships the source of a service to every node that runs it, which is rarely intended and occasionally a disclosure problem.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A multi-stage Dockerfile contains several
FROMinstructions. Each starts a new stage with its own base and its own layer stack. Only the final stage contributes layers to the published image. - A stage can copy selected paths out of an earlier stage with
COPY --from. That copy produces a new layer in the final stage containing only what was named — the rest of the earlier stage is discarded entirely, not deleted. - Stages are independently cacheable, and a builder that understands the graph can run independent stages in parallel and skip stages whose output nothing requests (Build Performance).
- Because the final stage picks its own base, the runtime's size, its libc, its shell and its scan surface are chosen separately from the build environment's. That decoupling is the real value; size is a consequence (What Image Size Actually Costs).
- The seam between the stages is where the classic failures live: the artifact is copied and something it depends on is not — a shared library, a CA bundle, timezone data, a config template, a non-root user entry.
Two stages, and what crosses between them
The only thing that reaches production is what the final stage explicitly copies. Everything else in the build stage — source, package caches, the compiler, any credential mounted for an instruction — is discarded rather than deleted, which is a stronger property.
The three extra copies in the final stage are the ones people forget, and each one produces a distinctive production failure the moment the service does real work.
1# syntax=docker/dockerfile:12 3FROM golang:1.23 AS build4WORKDIR /src5COPY go.mod go.sum ./6RUN --mount=type=secret,id=netrc,target=/root/.netrc go mod download7COPY . .8RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server9 10FROM gcr.io/distroless/static-debian12:nonroot AS runtime11# the binary12COPY --from=build /out/server /server13# the three things a minimal base does not have14COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/15COPY --from=build /usr/share/zoneinfo /usr/share/zoneinfo16USER nonroot:nonroot17ENTRYPOINT ["/server"]The secret mount is the part worth copying into your own builds: the credential exists for exactly one instruction and never becomes a layer in either stage. Note also the entrypoint is exec form — a shell form here would put a shell between the runtime and the signal, which is the subject of PID 1 and Signals, and this base has no shell to put there.
What each stage is responsible for
Reading the stages as having different jobs, rather than as one build split in two, is what makes the copy list obvious.
- 1Dependency stage
Resolves and downloads dependencies from pinned manifests.
fails by Unpinned resolution, or a credential written into a layer.
evidence A lockfile-driven install and no credential in any layer (Dependency Pinning).
- 2Build stage
Compiles or bundles into a known output path.
fails by Building against libraries the runtime base will not have.
evidence The output runs when started with the runtime base's libraries available.
- 3Test stage (optional)
Runs tests against the built output, targetable independently in CI.
fails by Testing the source rather than the artifact, so results do not describe what ships.
evidence Test results attached to the artifact identity (What an Artifact Is).
- 4Runtime base choice
Fixes size, libc, shell availability and scan surface.
fails by Chosen for size alone, leaving no diagnosis path (What Image Size Actually Costs).
evidence A written decision about how this image is debugged in production.
- 5Copy seam
Moves exactly the artifact and its runtime needs into the final stage.
fails by Missing CA bundle, timezone data, user entry, or a required shared library.
evidence The container starts, makes a TLS call, and runs as a non-root user.
- 6Entrypoint
Execs the process directly as PID 1.
fails by Shell form, which inserts a shell that may not forward signals (Graceful Shutdown).
evidence A termination signal reaches the application and it exits 0.
Single stage with cleanup, versus two stages
The left-hand form is what most single-stage Dockerfiles evolve into, and it is worth understanding exactly why the cleanup does not do what it appears to.
FROM golang:1.23 COPY . /src RUN go build -o /server /src/cmd/server RUN rm -rf /src /root/.cache /usr/local/go ENTRYPOINT ["/server"] shipped image contains: source, build cache, the Go toolchain, a shell, a package manager — all in earlier layers, all extractable
FROM golang:1.23 AS build COPY . /src RUN go build -o /out/server /src/cmd/server FROM gcr.io/distroless/static-debian12:nonroot COPY --from=build /out/server /server ENTRYPOINT ["/server"] shipped image contains: one binary
The cleanup on the left writes deletion markers into a new layer; the source, the cache and the toolchain remain in the layers beneath and are readable by anyone who pulls the image. On the right those layers are never part of the published image at all — the final stage starts from a different base and receives only what is copied. The difference is discarding versus hiding, and only one of them is a property you can rely on.
How to do it properly
Most important first.
- Name the stages.
AS build,AS deps,AS runtimemakes the copies readable and lets CI target a specific stage. - Copy the minimum from the build stage — the binary or the built assets — rather than a whole directory, so the runtime cannot accidentally inherit source or caches.
- Choose the runtime base for operability, not only for size, and pin it by digest (Tags Versus Digests).
- Verify what the built artifact actually needs before minimising the runtime base: dynamic library dependencies, CA certificates, timezone data, and the user the process runs as.
- Use the builder's secret mount for build-time credentials so they never become layers in any stage.
- Run as a non-root user in the final stage, and remember that a scratch or distroless base has no
/etc/passwdunless you supply one.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A broken runtime stage fails at start, so health checks stop the rollout before much traffic is affected (A Successful Deploy Is Not Evidence of a Healthy System). A credential shipped in a layer is contained by nothing — the image is distributed by design, and every copy has it.
What can go wrong
- A binary copied into a runtime base with a different libc, which fails instantly with a loader error and looks like a corrupt image (What Image Size Actually Costs).
- Missing CA certificates, so every outbound TLS call fails and the error points at the remote service.
- Missing timezone data, producing wrong timestamps or a crash on the first date formatting.
- A wildcard copy from the build stage that sweeps in the source tree and the package cache, defeating the point.
- Runtime configuration that lived in the build stage and was never copied, so the application starts and cannot find its templates or static assets.
- The mitigation failing: a runtime stage so minimal that the health check itself cannot run, because it was implemented as a shell command that now has no shell (Probes: Readiness, Liveness and Startup).
- "Multi-stage is a size optimisation." Size is a side effect. The point is that the build environment and the runtime environment stop being the same filesystem, which is what removes source, toolchain and credentials from the artifact.
- "The build stage does not matter because it is not shipped." It is where your dependencies are resolved and where build credentials are used. It is fully in scope for supply chain concerns (Securing the Pipeline Itself).
- "
COPY --fromcopies the layer." It copies files into a new layer of the final stage. Nothing about the source stage's layer structure carries over. - "If it builds, the runtime stage is correct." Building proves the compiler was happy. Whether the runtime base has the libraries, certificates and users the binary needs is only proven by starting it.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The published image contains no compiler, no package manager and no source, verified by inspecting the layers rather than by reading the Dockerfile.
- Extracting every layer of the published image finds no credential.
- The container starts, makes an outbound TLS call and formats a timestamp correctly — the three things minimal runtimes most often lack.
- Build times improve on changes that only touch the final stage, because the build stage is cached.
- A failed multi-stage change is reverted by deploying the previous digest; the change is entirely inside the artifact (Rollback: Only Useful If It Is Actually Safe).
- A credential that already shipped in a single-stage image is not fixed by moving to multi-stage. That image still exists in the registry and in every node's cache, so the response is rotation (Rotation That Applications Survive).
- Automate the layer inspection: fail the build if the published image contains a package manager, a compiler or a file matching a credential pattern.
- Automate stage targeting in CI so tests can run against the build stage without publishing it.
- Do not automate the choice of runtime base. It encodes a debuggability decision that belongs to whoever operates the service (What Image Size Actually Costs).
- Multi-stage builds are more Dockerfile to read and reason about, and the copy seam is a new place for a subtle omission.
- A different base for build and runtime means two bases to keep patched, and two sets of vulnerability findings.
- Very minimal runtime stages give the best size and security posture and remove the tooling an operator would use (Debugging a Container in Production).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALMulti-stage is an OCI-era Dockerfile feature supported by every mainstream builder, and the same separation is achievable in other build systems by building outside the image and copying the result in. The principle — the build environment is not the runtime environment — predates the syntax.
- PLATFORM-SPECIFICWhat the runtime stage must supply depends on the language. A statically linked Go or Rust binary can run on scratch with a CA bundle and timezone data; a JVM, Python or Node application needs its runtime present, so the final stage is a runtime base rather than an empty one, and "minimal" means a slim variant rather than nothing at all.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.