The question this answers
Why does capacity requested now start serving traffic minutes from now, and which part of that delay can I actually shorten?
A ticket sale opens at 10:00. Traffic goes from 200 req/s to 9,000 req/s in under thirty seconds. Any capacity that becomes available at 10:04 is capacity for a queue of angry users, not for the sale.
An honest budget for how long new capacity takes, broken into terms you can measure separately — which is the only way to know whether to shrink the image, pre-warm the pool, or give up on reacting and provision ahead.
Decompose the delay, then argue about it
"Startup time" is treated as one number and it is at least seven. Some of the terms belong to the platform, some to your image, some to your application, and some to the health check you wrote. Until they are separated, every conversation about scaling lag is opinion.
The terms compose differently for different compute models. A serverless function skips instance provisioning but pays runtime initialization on every cold invocation. A VM from a prebaked image skips image pull but pays a full guest-OS boot. A container on an already-running node with the image cached can be serving in a couple of seconds — and the same container on a fresh node pulling a 2 GB image is a minute of network transfer before a single line of your code runs. This is why Why Image Size Is an Infrastructure Problem is a *scaling* concern.
The uncomfortable finding is usually application warm-up: JIT compilation, connection pools opening, caches populating, a configuration fetch, a schema validation pass. It is the term nobody instruments and frequently the largest one.
- 1Metric window~60s
Load is aggregated over a period that must close before anyone can see it.
Irreducible without a faster metric pipeline; the floor on how early you can possibly react.
- 2Scaling decision~30–60s
The breach persists long enough to be believed, and a capacity request is issued.
Tightening this trades stability for speed — the flapping risk is real.
- 3Provisioning~20–40s
A host is chosen, network interface attached, storage attached, instance started.
Zone capacity errors and quota limits appear here, as a request that simply does not complete.
- 4Image pull~5–90s
The container image or machine image is fetched to the host and unpacked.
Scales with image size and registry distance. The one term you control most directly.
- 5Runtime boot~2–45s
Guest OS or runtime starts; the process is launched.
A full VM boot is far more than a container start — this is where the compute model shows.
- 6Application warm-up~5–60s
Connection pools open, configuration and secrets are fetched, caches fill, JIT compiles hot paths.
Usually the largest hidden term. A cold cache means the first requests hammer the database.
- 7Health check passes~15–30s
The readiness probe succeeds enough times for the load balancer to add the target.
Interval × threshold is a real delay you configured, and few people count it.
Red flag: "we scale when CPU hits 100%"
This is stated in interviews as though it were thrift, and it is the clearest possible signal that the speaker has never watched a scaling event. By the time CPU is pinned, the queue has formed, latency has already crossed the threshold users notice, and you are three minutes away from having any additional capacity at all. The scaling decision needs to be made while there is still headroom to survive the provisioning delay.
The arithmetic is simple and worth doing out loud. If new capacity takes 200 seconds and traffic grows 10% per minute, you must trigger at a utilization level that leaves at least 33% headroom, or the fleet saturates before help arrives. Targeting 100% is not aggressive; it is arriving after the fire.
The same reasoning kills the other common answer, "we will just set the threshold lower". Lowering the threshold buys you a fixed amount of time. Against a step function — a ticket sale, a push notification to two million devices, a scheduled batch — no threshold is early enough, because the load arrives faster than any reactive loop can respond. Step functions need capacity provisioned *before* the step: scheduled scaling, a warm pool, or over-provisioning for the known event.
time offered load fleet capacity state ------------------------------------------------------------- 10:00:00 200 rps 1,200 rps 4 instances, idle 10:00:20 9,000 rps 1,200 rps saturated; queue forming 10:00:40 9,000 rps 1,200 rps metric window still open 10:01:20 9,000 rps 1,200 rps breach confirmed -> scale out to 30 10:02:00 8,600 rps 1,200 rps users retrying; 504s at the edge 10:03:10 7,900 rps 4,500 rps first 11 instances healthy 10:04:30 6,100 rps 9,000 rps fleet complete; peak already passed 10:12:00 900 rps 9,000 rps 26 instances serving a trough Every instance that arrived was correct, and all of them were late. Scheduled pre-warm at 09:45 costs 15 minutes of idle fleet and removes the entire incident.
Shortening each term, and what each fix costs
Each term has a different lever, and the levers are not equally cheap. Shrinking the image is nearly free and helps every launch forever. Pre-warmed capacity works instantly and bills whether or not you use it. Provisioned concurrency for functions removes cold starts and removes most of the reason you chose functions.
Application warm-up deserves separate attention because it is the term teams most often mistake for a platform problem. A service that opens its database pool lazily will do so under the first burst of production traffic, at the exact moment the database is already busy. Opening the pool during startup — before the readiness probe passes — moves that cost to a moment when nothing depends on it. The same argument applies to loading a model into GPU memory, compiling templates, or fetching a large configuration blob.
And there is a term you can remove entirely: for a predictable event, do not react at all. Scheduled scaling has zero lag by construction, because the capacity exists before the load. It is unfashionable and it is correct.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Startup time is at least seven separate terms; treating it as one number makes it impossible to shorten.
- A reactive scaler cannot beat a step function. Predictable spikes need capacity provisioned before the step.
- "Scale when CPU hits 100%" guarantees the capacity arrives after the incident — the trigger must leave headroom for the provisioning delay.
- Image size is a scaling latency term, not just a build-hygiene concern.
- Application warm-up — pools, caches, JIT, model loading — is usually the largest hidden term and the one nobody instruments.
- Every fix that removes lag converts a usage-shaped cost into a fixed one. That is the trade, every time.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The scaler issues a capacity request; the platform allocates a host and attaches network and storage.
- • The host fetches the image — from a registry across the network on a cold node, from local disk on a warm one.
- • The runtime starts: a guest OS boot for a VM, a container runtime start for a container, a runtime init for a function.
- • The application initializes: configuration and secrets fetched, connection pools opened, caches primed, hot paths compiled.
- • The readiness probe begins succeeding; after the configured consecutive successes, the load balancer starts routing to the target.
- • Serverless platforms compress the first three steps and expose the rest as per-invocation cold-start latency on the tail of the distribution.
- • Instrument launch-to-first-request-served per instance and treat it as an SLI of the platform, not a curiosity.
- • Break the number down: log timestamps at process start, at pool-ready, at first successful readiness probe. Three timestamps end most arguments.
- • Keep launch images current, minimal and close by. A base image refresh is an operational task with a scaling-latency payoff.
- • Do warm-up before readiness passes, not after. Traffic on a cold process is a self-inflicted latency spike and often a database stampede.
- • Maintain a calendar of predictable events and pre-scale for them. This is unglamorous operational work that removes whole classes of incident.
- • Capacity arrives after the peak: correct decisions, useless timing, and a bill for instances serving a trough.
- • Image pull timeout on a cold node — the instance never becomes healthy and the group quietly stays under target.
- • Readiness passes before warm-up completes: traffic hits a cold process, first requests time out, and the target flaps in and out of rotation.
- • Warm-up stampede: fifty instances start together and all open pools and fill caches against the same database at once, so scaling out causes the database incident.
- • Cold-start tail on functions: p50 is excellent, p99 is thirty times worse, and only the users on the tail ever notice.
- • Zone capacity exhaustion during a large scale-out — the request for 40 instances returns 22 and no alarm fires.
- • Per-instance startup time is roughly constant; total time to reach a target grows with launch batch size and any per-launch rate limits.
- • A large simultaneous launch stresses the registry, the metadata service and the secrets store — components sized for steady state, not for a fleet-wide restart.
- • The larger the fleet, the more absolute capacity a fixed startup delay costs you during growth, so big fleets need pre-provisioning sooner.
- • For functions, cold starts scale with *concurrency growth*, not with request volume: a steady 10,000 req/s has few, a jump from 100 to 2,000 concurrent has many.
- • Warm-up fetches secrets. That path is on the critical latency path and on the critical trust path at once — it must be fast *and* authenticated with a workload identity (Roles vs Static Keys).
- • Pre-baked images shorten startup and age into a liability: an image built three months ago contains three months of unpatched packages.
- • Prefetching images to nodes means holding your application image in more places. Registry credentials and image digests matter more, not less (The Container Registry).
- • A long readiness path that touches many dependencies widens the window in which a half-initialized process is reachable on the network.
- • Every technique that eliminates cold start converts variable cost into fixed cost — warm pools and provisioned concurrency are pre-paid latency.
- • Frequent short-lived instances pay startup cost repeatedly: image pull bytes, provisioning overhead, and minimum billing increments.
- • Scheduled pre-scaling is usually the cheapest fix per unit of latency removed, because it only pays for the lead time.
- • Image size shows up on two meters at once: transfer bytes on every pull and the instance-minutes spent waiting for it.
- • Launch-to-healthy duration per instance, as a distribution rather than an average — the tail is the incident.
- • The stage breakdown: provision, pull, boot, warm-up, first passing probe. Without stages you can only guess.
- • Cold-start rate and cold-start latency for function workloads, tracked separately from warm invocations.
- • Time from scaling decision to fleet reaching target, which is the number that actually predicts whether you will survive the next spike.
- • The signal that lies: instance state showing "running". A running instance that has not warmed up is not capacity.
- • Scheduled scaling for anything predictable. No lag, no signal choice, no tuning — and for a ticket sale or a business-hours workload it is simply the correct answer.
- • Keep a modest permanent over-provision instead of engineering the delay away. For a small fleet, two extra instances are cheaper than a warm-pool mechanism plus its failure modes.
- • A queue in front of the work, so latency absorbs the spike and capacity never has to be fast. The best available answer for asynchronous work.
- • Admission control: shed or rate-limit excess load at the edge rather than trying to grow into it. A fast 429 is a better user experience than a 30-second timeout.
- • A platform that keeps capacity warm for you, accepting its pricing model as the price of not owning this problem.
- • Every second of startup removed is paid for in idle capacity, engineering time, or reduced deployment flexibility.
- • Aggressive pre-baking speeds launches and slows the security patch cycle.
- • Doing warm-up before readiness makes each instance slower to appear and much better behaved once it does.
- • Provisioned concurrency fixes the serverless cold-start complaint by removing the property that made serverless attractive.
Cold start, concurrency limit, and 1000 connections
What people believe, and what is true
Containers start instantly.
A container starts fast once its image is on the host. Pulling two gigabytes across a network before that is the majority of most cold launches.
Serverless removes the startup problem.
It relocates it into the request path as a tail-latency problem, where it is visible to users rather than to the scaler.
A lower scaling threshold fixes scaling lag.
It buys a fixed number of seconds. Against a step function nothing reactive is early enough — the capacity has to exist beforehand.