Capstone: Three Seconds from Warsaw

Three seconds is enormous. A single round trip Warsaw ↔ Frankfurt is ~20–30 ms; Warsaw ↔ US-East is ~100–120 ms; Warsaw ↔ Singapore ~180 ms. A well-built page from a distant region needs roughly: DNS (0–1 RTT to a resolver, more if uncached), TCP handshake (1 RTT), TLS 1.3 (1 RTT; TLS 1.2 is 2), the request and first byte of response (1 RTT plus server time), then the body at a rate that ramps with cwnd — perhaps 5–8 RTTs before the base document is in. At 100 ms that is 500–800 ms with a fast server. Three seconds means either many more round trips than necessary, a server that spends seconds, a loss-degraded path, or a page that needs dozens of dependent fetches.

The wrong answer is a list of technologies. The right answer is a budget: every layer named, in order, with a typical cost and a measurement, and then a bisection — measure the boundaries between layers until the missing seconds are found. The matrix below is the budget; the following section is the bisection. Each row names the Engineer Atlas domain whose lessons explain it, because the point of this capstone is that no single domain can answer the question.

Typical costs are for a 100 ms RTT path in a healthy state; they are orders of magnitude for reasoning, not measurements of your system. The measurement column is the tool that isolates that layer and nothing else — the property that makes a bisection possible. The lesson column is where to go when a row is the answer.

Read the domain column as a routing table for the investigation. Rows owned by Networking cost round trips and are fixed by removing round trips (caching, reuse, proximity); rows owned by Operating Systems cost waiting and are fixed by removing queues (capacity, pools, non-blocking I/O); rows owned by Database cost work and are fixed by doing less of it (indexes, fewer queries, smaller results). A three-second page usually has one row an order of magnitude over budget, and the domain tells you which kind of fix to expect before you have found it.

Start from the outside and split. First split: is it the network path or the server? curl -w from the user’s region against the same URL prints every phase — name lookup, connect, TLS, time-to-first-byte, total. If time_starttransfer minus time_appconnect (the server’s think time plus one RTT) is 2.5 s, the server is the problem and the network rows can be closed. If that gap is 150 ms but the total is 3 s, the time is in the transfer or before the request: check the size, the phases before it, and the number of requests.

Second split, server side: is it the request path or the dependencies? The server’s own timing — handler duration in its access log, or a trace — separates "we spent 2.4 s in the database" from "we spent 2.4 s in the queue before a thread picked it up" (pool wait, loop lag, accept-queue depth). The first leads to EXPLAIN ANALYZE and query counts; the second to top, run-queue length, and the The Thread Pool Server and The Event-Driven Server metrics. A server that is idle and still slow is either waiting on something remote (database, cache, a third-party API — measure each) or blocked on disk (iostat).

Second split, client side: is it round trips or bytes? Count them. The devtools waterfall shows how many connections were opened (each one is DNS + TCP + TLS unless reused — Keep-Alive and Connection Reuse, HTTP/2: Streams on One Connection multiplexing, Connection Pooling) and how deep the dependency chain is (a page that fetches a script that fetches a config that fetches data pays 4 sequential round trips before rendering). Bytes: log2(size/14 kB) round trips to ramp the congestion window means a 2 MB uncompressed bundle from 100 ms away is ~700 ms of pure cwnd growth before bandwidth matters at all. A CDN moves the RTT for static assets from 100 ms to 15 ms; nothing else moves it.

Each split closes half the matrix. Three or four splits reach a row, and the row’s lesson explains the fix. The discipline is to measure the boundary before naming the layer — the failure of most three-second investigations is that each team measures inside its own layer, finds it healthy, and stops.

A strong answer to the interview question (A user in Warsaw waits 3 seconds. Where did the time go?) names the layers in path order, gives an order of magnitude for each, states that 3 s cannot be explained by RTT alone on a healthy path from a fast server, and then proposes the bisection: curl -w from Warsaw first, then server-side timing, then either the database or the client waterfall. It mentions the things that silently multiply RTT — uncached DNS, no keep-alive, TLS 1.2, N+1 queries, sequential dependent fetches, a CDN miss, a cross-region database — because those are how 500 ms becomes 3 s without anything being "broken".

It also knows what each domain contributes and does not pretend one explains everything. The OS explains why an idle-looking server can still queue (scheduling, pool, loop lag, disk). Networking explains the round trips and the ramp. Database explains the query and the buffer cache. System Design explains the CDN, the balancer and the region choice. DSA explains why the handler took a second. And the answer ends with a number: "I would expect ~600 ms for this page from Warsaw; the missing 2.4 s is on the server, and I would find it with a trace".

The Capstone: A Server With 50,000 Concurrent Connections and Capstone: What Happens When You Visit https://example.com each ask the single-domain version of this question; Where the Time Goes: The Request Timeline and Latency: Same Machine to Cross-Continent give the networking numbers; Follow send() Through the OS to recv() gives the OS half of every network row. This lesson is the map that joins them.