ConfigGENERALPLATFORM-SPECIFIC

Build-Time and Runtime Configuration

When a value is fixed decides how you change it — build-time values need a new artifact and a full pipeline, runtime values change without one, and picking the wrong side quietly destroys build-once-promote-many.

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

Should this value be baked into the artifact or read when the process runs?

The problem

Some values are genuinely compiled in and cannot be changed later, some are read at every startup, and some can change while the process runs — and treating all three the same produces either environment-specific artifacts or values that cannot be changed when you need them changed.

What teams do first

Bake the values in at build time. It is simpler, there is nothing to configure at deploy, and the artifact is self-contained.

How it breaks

It makes the artifact environment-specific, so each environment needs its own build and the thing tested is not the thing shipped (Build Once, Deploy Many).

How it breaks in production
  • It makes the artifact environment-specific, so each environment needs its own build and the thing tested is not the thing shipped (Build Once, Deploy Many).
  • Changing a timeout becomes a full pipeline run. During an incident, "we need a rebuild to change that value" is a sentence that adds twenty minutes to a recovery.
  • It multiplies builds by environments, and each build is an opportunity for a supply-chain difference between what was tested and what ships (The Delivery Chain as Attack Surface).
  • It pulls secrets into images, where they persist in layers and registries indefinitely regardless of what the final image does (What Counts as a Secret, and Where It Must Not Be).
  • Meanwhile the opposite mistake is just as common: making everything runtime-changeable creates a large mutable surface that changes production instantly, with no review and no rollout (A Config Change Is a Production Change).
CodeBuildTestArtifactReleaseDeployRunObserveOperateIncidentRecoverLearnImprove

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

  • There are three binding times, and the distinction is when the value stops being changeable. Build time: compiled or bundled into the artifact; changing it requires a new artifact. Startup: read when the process boots; changing it requires a restart. Runtime: read on use or watched for change; it takes effect without a restart.
  • Each binding time has a different change cost and a different rollback story, and those two things are what you are actually choosing between.
  • Client-side bundles are the case that surprises people. Anything a browser bundle needs is baked at build time whether you intended it or not — an API base URL, a public key, an analytics identifier — unless the application fetches it at load from an endpoint the server renders (Preview Environments).
  • Runtime-changeable values require the process to re-read safely: a connection pool cannot resize mid-transaction, a cache size change may need eviction, and a value that some requests see and others do not creates a split within one process.
  • The related trap is the value read once at boot and cached forever, which looks like startup configuration and behaves badly when the underlying thing changes — this is exactly why rotation breaks applications (Rotation That Applications Survive).

Three binding times, three change costs

GENERALThe rows hold anywhere, but the boundary between the last two is language- and library-specific: whether a re-read applies to in-flight requests, pooled connections and long-lived streams depends on the client library, not on the configuration system.

Choosing a binding time is choosing a change cost and a rollback story. Writing them down together makes the choice concrete rather than a matter of taste.

Binding timeFixed whenTo change itRollbackGood for
Build timeCompilation or bundlingNew artifact, full pipelineDeploy the previous artifactCompiled constants; the minimum a client bundle needs
StartupProcess bootChange value, restart the fleetRestore value, restart againEndpoints, pool sizes, timeouts, secret references — the default
Runtime (polled or watched)Continuously re-readChange the value; it convergesChange it back; converges againLog level, sampling, feature state, kill switches
Runtime (per request)Every useImmediate on the next requestImmediateFlag evaluation, per-tenant behaviour (Feature Flags: Deploy Is Not Release)

The client bundle trap

This is the most common concrete way build-once-promote-many is lost, and it is usually not a decision — it is the default behaviour of a bundler that substitutes values at build time.

The consequence is a per-environment artifact, which means promotion is a rebuild and the bundle tested in staging is not the bundle users receive.

Getting an API base URL into a browser application
Substituted at build time
build for staging
  API_URL=https://api.staging...
  -> bundle-staging.js

build for production
  API_URL=https://api.example...
  -> bundle-production.js

  two artifacts, two builds
  promotion = rebuild
  a wrong value ships to every
    user and needs a full
    pipeline to fix
Fetched at load
one build
  -> bundle.js  (no environment
     values inside)

at load:
  GET /config
  -> { apiUrl, region, flags }
     served by the environment
     the app is running in

  one artifact, promoted
  a wrong value is a config
    change, fixable in seconds

The right column restores promotion for the client half of the system: one bundle is tested, promoted and served everywhere, and the environment-specific part becomes ordinary configuration with an ordinary rollback. The cost is one request before the application is usable, and a /config endpoint that must be available and cacheable — which is a smaller problem than a per-environment build (Promotion Between Environments).

When runtime configuration goes wrong

Runtime configuration adds a machine that can fail. These are the failures that actually happen, and they share a shape: the process keeps reporting healthy while running a value nobody believes it is running.

TriggerSymptomCauseResponse
Watch on the configuration store drops after a network blipOne instance serves a stale value indefinitely; health checks passThe watch was never re-established and nothing monitors watch healthExpose configuration version per instance and alert on non-convergence (Configuration Drift)
Pool size changed at runtimeConnections exceed the new limit for a long timeExisting connections are not closed by a size change; only new ones respect itDefine and test the re-apply path; treat pool changes as restart-scoped if it cannot be done safely
Value re-read per request mid-transactionOne logical operation uses two different valuesConfiguration read at several points inside one unit of workSnapshot configuration once per request or per transaction and use the snapshot throughout
Configuration store unavailable at startupFleet-wide failure to start, including services needed to fix itA hard startup dependency on a dynamic storeCache the last known good configuration on disk and start from it, logging loudly that it is stale
Secret cached at boot, rotated at the providerAuthentication failures begin abruptly, hours after any deployThe value was runtime-changeable in the store and effectively build-time in the process (Rotation That Applications Survive)Re-read on a bounded interval and on authentication failure
Kill switch itself stored behind the failing dependencyThe emergency control cannot be reached during the emergencyThe switch shares a failure domain with what it disablesKeep the kill switch on an independent path that does not require a redeploy or the failing dependency

How to do it properly

Most important first.

  • Default to startup configuration. It keeps the artifact environment-agnostic, validates cleanly, and has a well-understood change mechanism — a rolling restart (Validate at Startup, Fail Clearly).
  • Reserve build time for values that genuinely cannot be late-bound: compiled constants, and the small set a client bundle needs before it can call anything.
  • For client bundles, prefer fetching configuration from the server at load, so one bundle serves every environment and the deployment stays a promotion rather than a rebuild.
  • Reserve runtime for values that must change without a restart: log level, sampling rate, feature state, kill switches, and circuit-breaker thresholds (The Agent Kill Switch).
  • Make runtime re-reads explicit and safe. Define what happens to in-flight work when the value changes, and test that path rather than assuming it.
  • Never cache a secret for the life of the process. Re-read on a bounded interval or on authentication failure, so rotation does not require a deploy (Rotation That Applications Survive).
  • Record the binding time in the schema itself, so the change cost of any value is documented where people look.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone
One testEveryone
What contains it

Depends entirely on binding time, which is the lesson. A build-time mistake is contained by the deployment mechanism that ships the artifact — a canary catches it. A runtime change reaches every instance in seconds with nothing between it and all users unless the store supports staged rollout (Progressive Delivery: Exposure as a Dial).

What can go wrong

Failure modes, including of the mitigation
  • A build-time value discovered during an incident, when the fix requires a full pipeline run and nobody planned for that.
  • A browser bundle built with a staging API URL and promoted to production, which fails for every user immediately and cannot be fixed without a rebuild.
  • A runtime value that half the process respects — new requests use the new value, long-lived connections and pools keep the old one — producing behaviour that matches neither.
  • A configuration watch that silently stops after a network blip, so the process keeps serving a stale value indefinitely while reporting healthy.
  • A secret read once at boot and cached forever, which works perfectly until the first rotation (When Secrets Fail).
  • So many runtime-tunable values that the effective state of production is a set of console values nobody has reviewed together.
Misreads this invites
  • "Runtime configuration is strictly better because it is more flexible." It is more flexible and it is a larger mutable surface that reaches production instantly with no review. Flexibility is the benefit and the risk in one property.
  • "Environment variables are runtime configuration." They are read at startup in most runtimes. Changing one requires a restart, so they are startup configuration with a runtime-sounding name.
  • "The frontend can read environment variables." A browser bundle has no environment. Whatever was substituted at build time is in the file, and the only late-binding option is fetching it from the server.
  • "We can change it in the console, so it is runtime." Only if the process re-reads it. A value changed in a store that the process read once at boot has changed nothing until a restart — and that gap is where a rotation outage lives.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked
  • The same artifact digest runs in every environment, demonstrated by comparing what is deployed (Tags Versus Digests).
  • For any value, the schema states its binding time, so the change cost is knowable before the incident.
  • Changing a runtime value takes effect within a bounded, known time, verified by observation rather than assumed.
  • A rotation of a credential completes with no restart and no error spike.
How you get back
  • Build-time values roll back by deploying the previous artifact, which is clean but slow — a full pipeline if the previous artifact is not still available.
  • Startup values roll back by restoring the previous value and restarting, which is a rolling restart with its usual disruption (Draining: Stopping Without Dropping).
  • Runtime values roll back instantly, which is the strongest argument for putting kill switches on this side of the line — and instant change in both directions is also why an accidental change here reaches everyone immediately.
  • The awkward case is a value that changed binding time between versions. Rolling back the artifact can strand a value that the older version reads from a different place, or does not read at all.
What to automate, and what stays human
  • Automate the check that no environment-specific value is baked into the artifact — the same digest running everywhere is a testable property.
  • Automate the propagation and convergence check for runtime values, so a stalled watch is visible (Configuration Drift).
  • Do not automate the classification. Deciding whether a value should be changeable without a restart is a design decision that carries a blast radius, and it is exactly the judgement this lesson is teaching.
What this costs
  • Runtime configuration adds a dependency on a store and a code path that re-reads and re-applies values — machinery that can itself fail.
  • Startup configuration means every change is a rolling restart, which costs connection churn, cache warming and a window of reduced capacity (Headroom).
  • Build-time configuration gives the simplest, most auditable artifact and the worst change cost, which is a reasonable trade for a genuinely fixed value and a bad one for anything else.
  • Client bundles that fetch configuration at load add a request before the application is usable, which is a real latency cost on first paint.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

  • GENERALThe three binding times exist for any software. What varies is how many values are forced to build time: a compiled binary with constants and a browser bundle have a genuinely fixed set, while an interpreted server-side runtime can late-bind almost everything.
  • PLATFORM-SPECIFICServerless platforms typically fix environment variables at deploy time, so what is startup configuration elsewhere behaves as build-time configuration there — changing it is a deployment. Container platforms can update a mounted configuration file in place, which some runtimes will pick up without a restart and others will not, depending entirely on whether the application watches the file (ConfigMaps and Secrets).

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Domains that do not exist yet
  • Testing & Reliability Engineering — testing the re-apply path of a runtime configuration change, which is a code path most suites never exercise.