The question this answers
What belongs inside the image, what is injected at start, and where do secrets actually live?
The same checkout service must run against three databases, two payment providers and four log levels across development, staging and production — and the artifact that production runs must be the identical artifact staging tested.
A single promotable artifact whose behaviour is a function of injected configuration, so an environment change is a restart rather than a rebuild, and a compromised image does not hand over a credential.
One artifact, many environments
The rule is a small equation with large consequences: Application Image + Environment Configuration = Running Workload. The image contains everything that is the same everywhere — code, runtime, dependencies, sensible defaults. Configuration contains everything that differs — endpoints, credentials, feature flags, concurrency limits, log level. If an environment value is baked into a layer, you no longer have one artifact; you have three artifacts that happen to share a Dockerfile, and Build Once, Promote the Same Bytes stops meaning anything.
Configuration arrives in three shapes and they are not interchangeable. Environment variables suit small scalar values and are read once at start; they are visible in process listings and in orchestrator manifests, which makes them fine for a database *host* and wrong for a database *password*. Configuration files mounted at a path suit structured or large configuration and can be swapped without rebuilding. Secret references are the important one: the workload receives a pointer plus an identity, and fetches the value at start from a secret manager — so the value exists only in memory, can be rotated without redeploying, and every access is logged. See Secrets in Infrastructure and ConfigMap vs Secret — and the Honest Limit of a Secret.
A useful test: if changing this value requires a rebuild, it is in the wrong place. Environment configuration should be a restart, at most.
FROM python:3.11-slim
ARG STRIPE_KEY # recorded in the image config — readable by anyone
ENV DATABASE_URL=postgres://app:hunter2@prod-db.internal:5432/checkout
ENV STRIPE_KEY=${STRIPE_KEY}
ENV LOG_LEVEL=debug
COPY .env.production /app/.env # a credential file, now a permanent layer
COPY . /app
CMD ["python", "app.py"]
# consequences:
# - a separate image per environment; staging never tested these bytes
# - rotating the Stripe key requires a rebuild and a redeploy
# - the credential is in the registry, in every node's layer cache, foreverFROM python:3.11-slim
ENV LOG_LEVEL=info \
PORT=8080 \
HTTP_TIMEOUT_MS=3000 # safe defaults, identical everywhere
COPY . /app
USER 10001
ENTRYPOINT ["python", "app.py"]
# at start, injected by the platform — not present in any layer:
# DATABASE_HOST=prod-db.internal (env var: not sensitive)
# DATABASE_PASSWORD -> secret://prod/checkout/db (reference, resolved via workload identity)
# STRIPE_KEY -> secret://prod/checkout/stripe
# /etc/checkout/features.yaml (mounted file, swappable without rebuild)
# consequences: one artifact for every environment; rotation is a restart;
# an attacker who pulls the image gets code, not credentials.The image on the right is environment-agnostic, so the digest staging tested is the digest production runs. Rotating a credential no longer touches the build pipeline, and a registry compromise no longer implies a credential compromise.
A layer is forever, and it is trivially readable
The most important sentence in this lesson: an image layer cannot be un-published. Layers are additive and content-addressed, so a RUN rm in a later instruction records a whiteout and ships the bytes anyway (What Is Inside a Container Image). Build arguments are worse than files, because they are stored in the image *config document* — no filesystem archaeology required, just one command.
The demonstration below takes about ten seconds and needs nothing more than pull access. Every node in the fleet has that. Every CI job that ever built on top of this base has that. If the registry was ever public, or a token ever leaked, everyone has it.
The response to a baked secret is therefore not "rebuild without it". It is incident response: rotate the credential first, then rebuild, then work out how far the image travelled — mirrors, laptop caches, node layer caches, backups. Prevention is a build-time secret mount that never becomes a layer, plus a CI scan that fails the build when a credential appears in a layer or in the config.
$ docker history --no-trunc registry.example.com/checkout:1.14.2
CREATED BY SIZE
ARG STRIPE_KEY=sk_live_51H9x2eKq7Vd8mNpR4bTc 0B <-- build arg, in the config
ENV DATABASE_URL=postgres://app:hunter2@prod-db.internal:5432/… 0B <-- ENV, in the config
COPY .env.production /app/.env 412B <-- a layer
RUN rm /app/.env 0B <-- whiteout only
$ docker save registry.example.com/checkout:1.14.2 | tar -x -O --wildcards "*/layer.tar" \
| tar -x -O app/.env
STRIPE_SECRET_KEY=sk_live_51H9x2eKq7Vd8mNpR4bTc
DB_PASSWORD=hunter2
# The rm did not remove anything. The lower layer still ships the file.
# Everyone with pull access has had these credentials since the day it was pushed.Deciding where each value lives
Most configuration mistakes are placement mistakes rather than mechanism mistakes. The decision has three inputs: does the value differ between environments, is it sensitive, and does it need to change without a restart? Those three answers determine the home, and the table below is the whole decision.
One nuance is worth stating because teams get it backwards. A value that must change *without a restart* — a feature flag, a rate limit, a circuit-breaker threshold — does not belong in an environment variable at all, because environment variables are read once at process start. It belongs behind a configuration service or a flag system the application polls. Restarting the fleet to flip a boolean is a deployment, with a deployment's risk, for a change that was supposed to be cheap (Deployment Is Not Release).
| Value | Differs per env? | Sensitive? | Correct home | Cost of baking it into the image |
|---|---|---|---|---|
| Default log level, port, timeouts | No | No | Image ENV defaults | None — this is the right place. |
| Database host, queue URL, region | Yes | No | Environment variable at start | One image per environment; promotion becomes meaningless. |
| Database password, API key, signing key | Yes | Yes | Secret manager reference resolved via workload identity | Permanent disclosure to everyone with pull access; rotation needs a rebuild. |
| TLS certificate and private key | Yes | Yes | Mounted secret volume, or terminated at the load balancer | A leaked key that outlives every rotation you will ever do. |
| Structured config: routing rules, tenant map | Yes | No | Mounted configuration file | A rebuild for every routing change. |
| Feature flags, rate limits, thresholds | Yes | No | Flag or config service the app polls at runtime | A full deployment to flip a boolean. |
| Build metadata: commit SHA, build time | No | No | Image labels | None — labels are exactly for this. |
Key points
- Application Image + Environment Configuration = Running Workload. If a value differs per environment, it is not in the image.
- An image layer is permanent. A secret written into one has been published to everyone with pull access, and
rmin a later layer changes nothing. - Build arguments are recorded in the image config document —
docker historyreads them back with no filesystem work at all. - Environment variables suit non-sensitive scalars; sensitive values belong behind a secret-manager reference resolved by workload identity at start.
- A value that must change without a restart belongs in a flag or config service, not in an environment variable read once at boot.
- The response to a leaked secret in an image is rotate first, rebuild second, trace distribution third — in that order.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The image ships code plus non-sensitive defaults in its config document: env defaults, entrypoint, user, workdir.
- • At start, the platform merges deployment-supplied environment variables over those defaults and mounts any configuration files.
- • For secret references, the workload presents its identity to the secret manager and receives the value into memory — no credential is stored on disk or in the manifest.
- • The process reads configuration once at start; values that must change later are fetched at runtime from a configuration or flag service.
- • Rotation replaces the value in the secret manager; the workload picks it up on its next start or on its next fetch, with no image change.
- • Build-time secret mounts expose a credential to one instruction without recording it in any layer or in the config document.
- • You own configuration schema and validation at start. A workload that boots with a missing variable and fails on the first request has deferred a startup error into an outage.
- • You own the environment-to-value mapping and its review. Configuration changes reach production faster than code changes and get less scrutiny — that asymmetry is where outages come from.
- • You own secret rotation, including whether the workload can pick up a new value without a full redeploy.
- • You own CI scanning for credentials in layers and config, because prevention is the only cheap control here.
- • You own the audit trail of who read which secret, which the secret manager gives you and an environment variable does not.
- • A missing environment variable starts the container successfully and fails on the first request that needs it — a green deployment followed by a 100% error rate.
- • A staging endpoint is left in a production manifest, and production writes to the staging database for hours before anyone notices.
- • A rotated credential breaks every workload that cached it at start, because nothing was designed to re-read it.
- • A secret is baked into a layer and discovered months later; rotation is urgent, and the distribution trace covers mirrors, node caches and backups.
- • A configuration file mount shadows a directory that also contains application files, and the workload starts with parts of itself missing.
- • Environment variables leak into a crash log or an error-reporting payload, distributing the credential to a third party.
- • Environment variables scale badly past a few dozen values: no structure, no types, no validation, and a manifest nobody can read.
- • Secret fetches at start scale with replica count, so a large rollout produces a burst of secret-manager requests — usually rate-limited.
- • A configuration service adds a runtime dependency on the start path; if it is down, new replicas cannot start even though existing ones are fine.
- • What runs out first is human: an environment matrix wide enough that no one can state the difference between staging and production is already broken.
- • Never pass a secret as a build argument or an
ENVinstruction; both are recorded in the image config and readable with pull access. - • Prefer a reference resolved by workload identity over a value injected as an environment variable — it gives rotation, revocation and an audit trail.
- • Environment variables are visible in process listings, in orchestrator manifests, and often in crash dumps and error reporting.
- • Scope each secret to one workload and one environment; a shared credential makes revocation an outage decision instead of a routine one.
- • Scan images for credentials at build and on a schedule — see The Infrastructure Supply Chain and the Security domain's secret-lifecycle material.
- • Secret-manager requests are metered per operation on most platforms; caching the value for the process lifetime keeps that small.
- • Per-environment images multiply build minutes and registry storage by the number of environments — a direct cost of getting this wrong.
- • The real cost of a baked secret is the incident: rotation across every consumer, plus the audit of where the image travelled.
- • A configuration service is a fixed cost that replaces a variable one — rebuilds and redeploys for values that should never have needed them.
- • Startup configuration validation results, so a missing or malformed value fails loudly at start rather than quietly at request time.
- • Secret fetch success rate and latency, which is a start-path dependency and therefore a scale-out dependency.
- • A record of which configuration version each replica started with — the fastest way to explain why some replicas behave differently.
- • Secret-manager access logs: which identity read which secret, when. That is the audit trail an environment variable cannot give you.
- • The signal that lies: a successful rollout. It proves the containers started, not that they were given the right configuration.
- • A plain mounted file with restrictive permissions, injected by the platform, when a full secret manager is more machinery than the team needs. Far better than baking it in.
- • Terminating TLS at the load balancer so the certificate and key never reach the workload at all — the simplest way to remove an entire class of secret.
- • For a single-environment internal tool, a
.envfile that is deliberately never committed remains a defensible answer; the failure mode is committing it, not using it. - • Provider-native workload identity for provider services, which removes the credential entirely: no key exists to bake, leak or rotate. Prefer this wherever it is available — see Roles vs Static Keys.
- • External configuration buys one promotable artifact and costs a new start-path dependency that must be available for new replicas to boot.
- • Secret references buy rotation, revocation and audit; they cost a secret manager, a workload identity setup, and latency on every cold start.
- • Environment variables buy simplicity and cost structure, validation and any protection for sensitive values.
- • A runtime configuration service buys change without redeployment and costs a component whose outage prevents scale-out.
The lifecycle of one database credential
| If it leaks | Exposure window | How you find out |
|---|---|---|
| static credential, no expiry full database access from anywhere on the internet that can reach the endpoint | until someone notices and dares to revoke it — typically months | a scanner, a bill, or a customer |
| short-lived token, 1 hour the same access, for one hour, from one identity that the log names | at most 60 minutes, then it is refused with no action from you | the issuing log shows the unexpected request |
What people believe, and what is true
Deleting the file in a later instruction removes the secret.
It records a whiteout. The bytes are in a lower layer and still ship on every pull. Rotate the credential; the image cannot be fixed retroactively.
Build arguments are safe because they are not in the filesystem.
They are stored in the image config document, which is exactly what docker history prints. They are the easiest secret to extract from an image.
A private registry makes baked secrets acceptable.
Every node, every CI job and every mirror has pull access. Private means access-controlled, not secret.
Environment variables are a secure way to pass credentials.
They are better than a layer and worse than a reference: visible in process listings, in manifests, and often in crash dumps and error reports.