Latency, Throughput & Queueing
The physics of a loaded system: tail latency, latency budgets, Little's Law as working intuition, why queueing makes systems slow long before they fail, and what saturation actually means.
Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.
The dashboard says 120 ms and users say it is slow. Both are right: "the latency" was never one number. Response time decomposes into service time and wait time, and almost every production surprise lives in the waiting.
A one-in-a-hundred slow response sounds harmless until a page makes 40 calls, a user makes 30 page views, and every dependency has its own one-in-a-hundred. Rare events compose, and at scale the tail becomes the typical experience.
A latency target is only useful once it is divided. Give every hop an allowance, add them up, and the conversation changes from "make it faster" to "the payment provider is spending 45% of our budget and we have 15 ms left".
"We handle 10,000 requests per second" is not a capability claim until you say at what latency and with what error rate. Push a system to its maximum throughput and you will find the maximum is a place nobody wants to operate.
Concurrency equals throughput times latency. Three lines of arithmetic size a connection pool, expose an impossible capacity claim, and turn a queue depth into a wait time — which is most of what the law is for.
Load rises 20% and latency rises 400%. Nothing errored, no code changed, no dependency degraded. A queue formed — and queues turn a linear increase in arrivals into a non-linear increase in waiting.
Utilization says how busy a resource was. Saturation says how much work could not be served immediately. A CPU at 55% with twelve threads waiting for eight cores is not half idle — it is oversubscribed, and only one of those two numbers says so.
Accepting every request that arrives feels generous and produces the worst possible outcome: everything is slow, everything times out, and the capacity is spent on work nobody is still waiting for. A limit is a latency control, not just a safety valve.
A timeout is a statement about how long a caller will wait before deciding the answer is worthless. Set it too short and you manufacture load; too long and you tie up capacity waiting for work that stopped mattering minutes ago.