The question this answers
What does an extra gigabyte of image actually cost, and where does that cost land?
The API must add capacity within ninety seconds of a traffic spike. Today the alarm fires, an instance appears in forty seconds — and then spends two more minutes pulling a 2.1GB image while the existing replicas absorb the overload.
A deployable artifact small enough that pulling it is not the dominant term in time-to-ready, and narrow enough that it does not ship a compiler, a package manager and a shell for an attacker to find.
Four separate costs, one cause
Image size is usually filed under tidiness. It is not: it is a latency budget, a bill and an attack surface at the same time. Pull time delays every cold start, and a cold start is what an autoscaler produces by definition. Deployment time stretches every rollout, which stretches the window in which two versions serve traffic. Transfer bills through the egress or NAT meter on every pull, multiplied by replica count and deploy frequency. Attack surface grows with every package present: a shell, curl, a package manager and a compiler in the runtime image are four tools an attacker no longer has to bring.
The four are not equally visible. Storage is cheap and everyone notices it; the transfer meter is invisible until the bill arrives; and the autoscaling delay is almost never attributed to the image, because the graph that shows it is the request-latency graph, not the image graph.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Multi-stage: build in one image, ship from another
Nearly every oversized image has the same cause — the tools that *built* the artifact are still inside the artifact. A compiler, header files, a test framework, a package cache and a .git directory are all necessary at build time and all useless at run time, and because layers are additive, deleting them later does not remove them (What Is Inside a Container Image).
A multi-stage build resolves this by making the build environment a different image from the shipped one. The first stage installs whatever it needs and produces a compiled binary or an installed dependency tree. The final stage starts from a minimal base and copies only that output across. Nothing from the build stage ships unless it was explicitly copied — the toolchain is not deleted, it was never in the artifact.
The extreme version is a static binary on a scratch or distroless base: a handful of megabytes, no shell, no package manager, nothing to patch. It is worth naming the cost honestly. No shell means no shell during an incident. Debugging shifts to ephemeral debug containers and to good telemetry, and a team without either will feel that loss at the worst possible moment.
FROM golang:1.22 # ~800MB: compiler, stdlib source, build cache WORKDIR /src COPY . . # includes .git, tests, fixtures RUN go build -o /app/server ./cmd/server EXPOSE 8080 CMD ["/app/server"] # result: ~1.1GB, of which ~12MB is the program that runs. # ships: a Go compiler, a package manager, a shell, curl, the full source.
FROM golang:1.22 AS build WORKDIR /src COPY go.mod go.sum ./ RUN go mod download # cached until the manifest changes COPY . . RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/server ./cmd/server FROM gcr.io/distroless/static:nonroot # no shell, no package manager COPY --from=build /out/server /server # the only thing that crosses USER nonroot:nonroot ENTRYPOINT ["/server"] # result: ~15MB. Pull time stops being the dominant term in time-to-ready. # cost: no shell for `exec` debugging — you need ephemeral debug containers.
Same program, roughly seventy times less to move on every pull, and an attacker who lands inside finds no shell, no package manager and no compiler. The price is that you can no longer debug by opening a shell in the running container.
The cost nobody attributes: new capacity arrives late
Autoscaling is always reactive. A signal crosses a threshold, a decision is made, an instance is provisioned, the image is pulled, the process starts, health checks pass, and only then does traffic arrive. Image pull sits in the middle of that chain and is frequently the largest single term — and unlike instance boot, it scales linearly with a number the application team controls.
The incident shape is specific. Traffic doubles in ninety seconds. The autoscaler reacts in twenty. The instance is ready to pull in forty. Then it downloads 2.1GB across a NAT gateway at whatever throughput it gets, taking two and a half minutes, while the existing replicas queue requests, exhaust their connection pools and start returning 503s. The postmortem says "autoscaling was too slow" and the remedy is usually a higher minimum replica count — permanent idle capacity purchased to compensate for an artifact that was never trimmed. See Startup Time & Cold Start and Autoscaling Signals.
Two structural mitigations exist besides shrinking the image: pre-pull the image onto nodes before it is needed, and share base layers across services so a node that already runs anything has most of the new image cached. Both help. Neither helps as much as not shipping the compiler.
| Image | Uncached pull | Process start | Health checks | Total time-to-ready | Consequence |
|---|---|---|---|---|---|
| 2.1 GB (single-stage, full toolchain) | ~150 s | ~8 s | ~10 s | ~168 s | Scale-out finishes after the spike has already caused errors. |
| 420 MB (slim base, deps pruned) | ~30 s | ~8 s | ~10 s | ~48 s | Usable, still the dominant term in the budget. |
| 15 MB (multi-stage static binary) | ~2 s | ~1 s | ~10 s | ~13 s | Health checks become the dominant term — the right place for it to be. |
| Any size, layers already cached on the node | ~0 s | as above | as above | start + checks | Why pre-pulling and a shared base pay off across a fleet. |
Key points
- Image size is four costs at once: pull latency, deployment duration, transfer billing and attack surface.
- The expensive one is usually scale-out lag, and it is almost never attributed to the image in a postmortem.
- Multi-stage builds work because the toolchain is never copied into the final artifact — not because it is deleted afterwards.
- A minimal base removes real attack surface and removes your shell; budget for ephemeral debug containers before you commit to it.
- Layer sharing and pre-pulled images make a cold node warm; a common base across services is a fleet-wide latency and cost improvement.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A pull fetches every layer the host does not already have, decompresses it and writes it into the local layer store.
- • Layers are fetched in parallel but the container cannot start until all of them are unpacked, so the slowest layer sets the floor.
- • Content addressing means a layer shared with an image already on the node is skipped entirely — the basis of every pre-pull strategy.
- • A multi-stage build discards intermediate stages: only paths named in a
COPY --frombecome layers in the final image. - • Compression matters at transfer time and not at start time; a highly compressible layer still costs full decompression on the node.
- • You own a size budget per service, checked in CI. "The image grew 400MB" should fail a build, not surface six months later.
- • You own
.dockerignore. Shipping.git,node_modulesand test fixtures into the build context is the most common silent bloat. - • You own base image standardization across services, which is what makes layer sharing actually happen on a node.
- • You own the debugging story once the shell is gone: ephemeral debug containers, or good enough telemetry that you do not need one.
- • You own registry retention. Old large images cost storage and widen the set of artifacts something might accidentally pull.
- • Autoscaling appears broken: capacity is provisioned on time and still arrives after the spike, because pull time was never in anyone's model.
- • A rollout stalls because a hundred nodes pull the same large image simultaneously and saturate the NAT path — a self-inflicted thundering herd.
- • A registry rate limit is hit mid-rollout; half the fleet runs the new version, half cannot pull, and the deployment is stuck in a mixed state.
- • A node runs out of disk from accumulated image layers, and every subsequent pull on that node fails with no space left on device.
- • A vulnerability scan reports two hundred findings, all in build tooling that only exists in the image because the build was single-stage.
- • Pull cost scales with replicas × deploy frequency × node churn — a fleet that triples triples the transfer bill from the same image.
- • Cold-start cost scales linearly with size, so image size directly sets the floor on autoscaling responsiveness.
- • Layer sharing scales in your favour: standardizing on one base makes the marginal image on a node much smaller than its nominal size.
- • What runs out first during a large rollout is registry throughput or NAT bandwidth, not node CPU.
- • Every binary in the image is a tool an attacker can use post-compromise. A shell,
curl,wgetand a package manager are the standard first four. - • Build tooling in a runtime image means compilers and credentials-adjacent utilities in production, and a much longer vulnerability list.
- • Smaller images are faster to scan and produce findings you can act on rather than a wall of base-image noise.
- • Distroless and static images trade incident ergonomics for a much smaller surface. Make that trade knowingly — see Container Security and Its Limits in the Security domain.
- • The dominant meter is transfer, not storage: bytes × pulls, and pulls are far more frequent than most teams estimate.
- • Pulls that cross a NAT gateway, a zone boundary or a region boundary bill again on those meters.
- • Slow scale-out is paid for with permanently higher minimum capacity — the most expensive way to fix a large image.
- • Registry storage is real but small; optimizing it first is optimizing the wrong item.
- • Image pull duration per node, ideally as a histogram — the number that tells you whether the artifact is in your latency budget.
- • Time-to-ready broken into provision, pull, start and health-check phases, so scale-out lag is attributable.
- • Image size per build over time, with a CI gate on regression.
- • The signal that lies: instance count. The autoscaler reports the new instance as added while it is still downloading and serving nothing.
- • Change the base image and prune dependencies before restructuring the build. Switching to a slim base is often a one-line change that removes most of the excess.
- • Pre-pull or cache images on nodes if the image genuinely cannot shrink — common for ML images with large model or CUDA layers.
- • Keep a larger image and raise the minimum replica count if the workload is not spiky. Paying for headroom is a legitimate choice when scale-out latency does not matter.
- • For a batch job that runs nightly, image size is nearly irrelevant. Optimize it only where cold-start latency or pull volume actually bites.
- • A minimal image buys pull speed and a small attack surface; it costs the shell, the package manager and easy in-container debugging.
- • Multi-stage builds buy a small artifact and cost a more complex Dockerfile that a newcomer will find harder to modify correctly.
- • A shared standardized base buys layer reuse across the fleet and costs coordination — every service now depends on one team's base image cadence.
- • Aggressive stripping buys size and can cost observability: symbols removed with
-s -ware symbols missing from your stack traces.
What people believe, and what is true
Image size only matters for storage.
Storage is the cheapest of its four costs. Pull latency, transfer billing and attack surface all cost more.
Adding a cleanup step at the end shrinks the image.
Layers are additive. Files removed in a later layer are still shipped; only a separate build stage keeps them out.
Autoscaling lag is an autoscaler configuration problem.
Often it is the image. Decompose time-to-ready before touching thresholds and cooldowns.