Timeouts
Never assume an external dependency returns. A call with no timeout is a resource leak waiting for a bad day.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How do I decide how long to wait for something outside my process — and what happens if I never decide?
Checkout calls a payment provider. If the provider is having a bad minute, checkout must fail cleanly rather than take the service down with it.
Await the call. The library will surely give up eventually, and if it does not, the request will error out somehow.
It does not error out. A connection that is open but silent produces no packets and no exception; the await simply never resolves, and the worker holding it is gone for good.
- It does not error out. A connection that is open but silent produces no packets and no exception; the await simply never resolves, and the worker holding it is gone for good.
- The number of stuck requests grows monotonically. Nothing recovers, because nothing has a mechanism to recover — you have a leak with a socket attached (Connection Pool Exhaustion).
- Your caller's timeout fires first, so the user sees a 504 while your handler keeps working, still holding a database connection for a client that has left (The Request Lifecycle).
- A restart appears to fix it, which teaches the team the wrong lesson and hides the actual defect for months (Memory Leaks in Backend Services has the same diagnostic shape).
- The timeout you did set was per-attempt, and with three retries the real worst case is more than three times what you budgeted (Retries).
What is actually happening
- TCP has no built-in application-level deadline. A peer that accepts your connection and then says nothing produces silence, and silence is indistinguishable from "still working" (The Connection Lifecycle: Close, Reset, TIME_WAIT, CLOSE_WAIT in Networking covers the socket side).
- A call without a timeout is not "slow" — it is unbounded, and unbounded consumption of any finite resource is a leak. The resource is a worker, a connection, a file descriptor and a slot in whatever pool you did not think about (Connection Pools).
- Most clients expose several timeouts that mean different things: connect, TLS handshake, time-to-first-byte, read-idle, and total. A 5-second read-idle timeout on a slow trickling response never fires while the total elapsed time grows without limit.
- A timeout is a local decision to stop waiting. It says nothing about the other side, which may have completed the work, may still be doing it, and will never know you left (Retries).
- Timeouts compose along a chain. If your caller allows 2 seconds, every dependency deadline you set inside that must fit within what remains — a deadline propagated down the chain is the correct model, and a fixed per-call timeout is an approximation of it (Deadlines vs Timeouts in Concurrency).
- A timeout that only abandons the wait, without cancelling the underlying work, leaks the connection anyway. Firing a timer is not the same as freeing the resource.
How to choose the number
There is no universal timeout value, and any advice that offers one is describing someone else's dependency. What is universal is the procedure, which takes two inputs and resolves the conflict between them in a fixed direction.
The first input comes from the dependency: measure the latency of its successful calls and look at the high percentiles. Including failures pollutes the distribution with the very timeouts you are trying to set. The second input comes from you: the deadline your caller gave, minus what this request has already spent, minus what the remaining work needs. Whichever is smaller is your ceiling — and if the dependency's tail does not fit inside your budget, the honest conclusion is that this call does not belong in the request path.
- 1Measure successful-call latency
Build a histogram of durations for calls that returned a real answer.
fails by Using an average, which hides the tail entirely, or including timed-out calls, which is circular.
- 2Pick the tail you intend to serve
Decide which percentile of legitimate slow calls you are willing to wait for.
fails by Choosing the median, so the timeout fires constantly on normal traffic.
- 3Compute the remaining budget
Caller's deadline minus elapsed minus the cost of the rest of the request.
fails by Never being computed, so per-call timeouts sum past the caller's limit and become decoration.
- 4Take the smaller of the two
The budget is a hard ceiling; the distribution is a preference.
fails by Letting the dependency's needs override your own deadline, which just moves the failure upstream.
- 5Fit retries inside it
Total attempts plus backoff must not exceed the deadline (Backoff and Jitter).
fails by Per-attempt timeouts multiplied by attempt count, blowing the budget silently.
- 6Re-derive when reality changes
Re-measure after vendor changes, region changes and traffic growth.
fails by A constant chosen in year one, still in the config in year three, matching nothing.
Notice that no step in this procedure produces a number you could have guessed. That is the point.
Setting the bound where it actually binds
asyncio.timeout. Some ecosystems propagate cancellation automatically through the call tree; in others every layer must pass it explicitly, and any layer that forgets breaks the chain.The most common way to have a timeout and still hang is to bound the wrong thing. A read timeout bounds the gap between bytes, not the whole response; a connect timeout bounds the handshake, not the call; and neither of them bounds waiting for a connection from an exhausted pool.
The version below bounds the total operation and, critically, aborts it — which is what releases the socket. A timer that resolves your promise while the request keeps running has bounded your patience and nothing else.
const res = await fetch(url, { method: 'POST', body })
// no timeout at all: a silent peer holds this worker forever
// or, only marginally better:
const res = await Promise.race([
fetch(url, { method: 'POST', body }),
sleep(2000).then(() => { throw new Error('timeout') }),
])
// the throw abandons the await; the request keeps running
// and the connection is never releasedasync function callProvider(body: unknown, deadline: Deadline) {
const remaining = deadline.remainingMs() // budget, not a constant
if (remaining <= 0) throw new DeadlineExceeded() // do not even start
const ac = new AbortController()
const timer = setTimeout(() => ac.abort(), Math.min(remaining, PROVIDER_MAX_MS))
try {
return await fetch(url, {
method: 'POST',
body: JSON.stringify(body),
signal: ac.signal, // cancels the request, frees the socket
headers: { 'Idempotency-Key': body.opId }, // the retry is now safe
})
} catch (e) {
if (ac.signal.aborted) throw new TimeoutError({ waitedMs: remaining })
throw e
} finally {
clearTimeout(timer)
}
}The abort signal is the difference between stopping the wait and stopping the work: without it the socket stays open and the leak continues even though your code took the error path. Deriving the bound from the remaining deadline rather than from a constant is what keeps a chain of calls inside the budget the caller actually granted.
What a timeout does not tell you
A timeout is the most ambiguous outcome in backend engineering. Three completely different states produce exactly the same observation, and no amount of client-side cleverness distinguishes them.
This is why the answer to "should I retry a timeout?" is never a property of the timeout. It is a property of the operation: if the operation is idempotent, retry; if it is not, either make it idempotent with a key or reconcile afterwards (Idempotency in Backends).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Request never arrived | Timeout. | Connection or network failure before the peer read anything. | Safe to retry. Nothing happened on their side. |
| Request arrived, still processing | Timeout. | The dependency is slow, not broken; the work may yet complete. | A retry runs it a second time concurrently. Only safe with an idempotency key (Idempotency Keys). |
| Request completed, response lost | Timeout. | The effect happened; the acknowledgement did not reach you. | A blind retry duplicates the effect. Key it, or reconcile against their records. |
| Timeout fired, work not cancelled | Timeout, and the in-flight count never drops. | The timer resolved the promise but nothing aborted the request. | Cancel on timeout; alert on in-flight counts that do not return to baseline. |
| Pool wait exceeded the timeout before the call started | Timeout with near-zero dependency latency in their logs. | The bound did not include connection acquisition (Connection Pools). | Bound the whole operation; measure acquisition separately. |
| Timeout longer than the caller's | Client sees 504; your logs show the call succeeding afterwards. | No deadline propagation; each layer chose independently. | Propagate an absolute deadline down the chain (Request Context Propagation). |
How to build it
Most important first.
- Set a timeout on every outbound call. No exceptions, including calls to services you own, calls to your own database, and calls the SDK makes on your behalf.
- Choose the value from two inputs, not from a habit. First, the dependency's observed latency distribution for successful calls — measure it, look at the high percentiles, and pick a value comfortably above the tail you intend to serve. Second, your own remaining budget: what your caller allows, minus what you have already spent, minus what the rest of the request still needs.
- When the two disagree, the budget wins. If a dependency needs longer than your request can afford, that call does not belong in the request path — it belongs in a job (Request or Background?).
- Carry a deadline, not a duration. Pass the absolute time the request must finish through the call chain so each hop can compute what it has left, and so retries cannot exceed the total (Request Context Propagation).
- Set the timeout at the level that actually bounds the whole operation. Prefer a total-operation deadline over a read-idle timeout, and check whether your client's timeout includes connection acquisition from the pool — frequently it does not.
- Make cancellation real: abort the request, release the connection, and stop the downstream work where the protocol allows it (Cancellation Propagation in Concurrency).
- Treat a timeout as its own error class, distinct from a failure response, because the outcome is ambiguous and the correct handling differs (An Error Taxonomy That Maps Cause to Response).
What can go wrong
- A timeout set on the HTTP read but not on connection acquisition, so a saturated pool queues indefinitely before the timer ever starts.
- A per-attempt timeout multiplied by retries, so a "2 second" dependency has a real worst case of 6 seconds plus backoff.
- A timeout longer than the caller's, which makes it decoration — the caller gives up first and your work is orphaned.
- Timeout tuned to the p99 of a healthy day, so it fires on every request during a degraded one, converting a slowdown into a total outage.
- A timeout so aggressive it abandons work that was about to succeed, at which point the retries you added to compensate double the load (Retry Storms).
- Timeout fires, the vendor completes anyway, the retry charges the card a second time (Idempotency Keys).
- A background job with no timeout at all, because "it is not in the request path" — so it holds a worker forever and the queue stops draining (Queue Backlog).
- The response arriving in the same instant the timer fires: the work happened, you recorded failure, and any retry is a second execution (Duplicate Detection).
- Timeout firing while the connection is being returned to the pool, so a cancelled call's socket is reused for the next request if cleanup is not correct (Connection Pools).
- A deadline computed once and reused across retries by two concurrent code paths, so one path's cancellation aborts another's work.
- A dependency that never returns is an availability vulnerability an attacker can trigger deliberately: any input that steers a request toward a slow path becomes a denial-of-service primitive.
- Slow-response attacks work in both directions. A hostile or compromised dependency that trickles bytes indefinitely can exhaust your workers, which a total-operation deadline prevents and a read-idle timeout does not (What Happens When the Receiver Is Slow in Networking).
- Timeouts on user-supplied destinations are a required part of SSRF defence — without one, an attacker points you at a black hole and holds your capacity (SSRF — When the Backend Fetches a URL).
- "The library has a default." Sometimes. Several widely-used HTTP clients default to no timeout at all, and several others default to a value far above any sensible request budget. Read the specific client; do not generalise.
- "Set it to 300 ms" — or any fixed number offered as universal advice. The right value is a function of that dependency's measured distribution and your remaining budget, and it differs per dependency, per operation and per environment.
- "A timeout means the call failed." It means you stopped waiting. The operation may have completed — which is why the retry question is really an idempotency question (Retries).
- "Longer is safer." Longer means holding a worker longer, which makes a partial degradation into a full one faster.
- "Only external calls need timeouts." Your database, your cache, your own internal service and your DNS resolver are all across a process boundary and all capable of not answering.
- "We have a timeout" — check which one. A read-idle timeout does not bound a response that trickles a byte at a time, and a connect timeout does not bound anything after the connection opens.
Operating it
- A timeout counter per dependency, separate from the error counter. A rising timeout rate with a flat error rate is the earliest signal of a degrading dependency.
- A client-observed latency histogram that includes connection acquisition, and read it as a distribution rather than an average (Percentiles: Which One, and How Many Users Is That? in Performance).
- In-flight call count per dependency. Monotonic growth is the leak, visible long before the pool is empty (Unbounded Concurrency).
- The ratio of your timeout to the dependency's observed p99 for successful calls. When that ratio drifts toward 1, the timeout is about to start firing constantly.
- Log the elapsed time and the remaining deadline on every timeout. "Timed out after 1.9s with 0.1s of budget left" is a different problem from "timed out after 1.9s with 8s left".
- At 10x, tail latency at the dependency rises even when the median does not, so a timeout sized against the median starts firing on real traffic (Tail Latency: Why p50 Being Fine Does Not Help).
- At high concurrency, connection acquisition becomes a meaningful part of total time, and a timeout that excludes it stops bounding anything useful.
- Aggressive timeouts plus retries interact badly at scale: the abandoned work is still executing on the dependency, so you are adding load while reducing your own useful throughput (Little's Law as Working Intuition in Performance explains the mechanism).
- Deadline propagation matters more the deeper the call chain: without it, a five-hop chain compounds five independent guesses into a number nobody chose.
- Short timeouts fail fast, protect the pool and abandon work that would have completed. Every one of those abandonments is a user-visible failure you selected on purpose.
- Long timeouts preserve success rate under mild degradation and consume capacity during severe degradation, which is exactly when you have none to spare.
- Deadline propagation is strictly better and costs plumbing through every layer, every client and every library that does not support it.
- A total-operation deadline is the correct bound and is not always available: some clients only offer connect and read timeouts, and building a real one means wrapping the call.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALEvery call leaving the process needs a bound, in every language and runtime. The requirement is absolute even though the number never is.
- FRAMEWORK-SPECIFICClient semantics differ materially: some expose one total-request deadline, others expose separate connect/read/write timeouts that can sum well past your intent, and some exclude pool-wait time from all of them. Verify what the specific client bounds before trusting the number.
- RUNTIME-SPECIFICWhat a timeout must clean up differs. A thread-per-request runtime needs the blocked thread released; an async runtime needs the task cancelled and the socket closed, and an abandoned promise with no cancellation keeps the connection open even though your code has moved on (Backend Runtime Models).
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — why a timeout cannot distinguish a slow peer from a dead one, and what that impossibility forces into the application.