Failure Models

A Timeout Tells You Nothing About Whether It Happened

Service A calls Service B and the call times out. The single most common mistake in distributed systems is treating that as "it failed". It is not a failure result — it is the absence of a result, and five different realities produce it.

▶ Run the lab

The question this answers

The question

My request to another service timed out. Did the work happen?

The guarantee — the property claimed, and its scope

None. A timeout is the absence of information, not a negative result. The only sound claim after a timeout is "the outcome is unknown to me", and any correctness argument must hold for both outcomes.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A knows exactly one fact: no response arrived within the deadline it chose. A does not know whether the request was received, whether B executed it, whether B committed, or whether B replied. Every other belief A holds about B is inference — and the timeout is precisely the case where that inference is unavailable.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?→What does a node know?→How does it work?→What can fail?→How does it fail?→Where is coordination?→What holds under failure?→How does it recover?→How would you know?→What is the simpler thing?
timeoutpartial failureambiguityretries

Five realities, one observation

A sends POST /charge to B and waits two seconds. Nothing comes back. Here is the complete list of what may have happened, and the crucial property they share: A cannot distinguish them from where it stands.

Notice that three of the five involve the work having been *done*. Treating a timeout as failure is not a conservative assumption — it is wrong more often than it is right for any service that mostly works.

The same observation at A, five different truths at Bprotocol
Service AService BPOST /charge: deliveredPOST /charge200 OK: sent, never arrives — dropped in flight200 OKdropped — never arrivessend POST /charge (write) at t=0send POST /chargecharge committed (write) at t=3charge committeddeadline expires — no information (decide) at t=8deadline expires — no informationt=0time →t=8
delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswritedecide
The response was lost, so the card was charged and A believes nothing happened. Redraw this with the *request* dropped and A observes exactly the same thing.

Why the ambiguity is not removable

It is tempting to look for a protocol that removes the doubt — an acknowledgement of the acknowledgement. It does not help: the new ack can itself be lost, and you have moved the ambiguity rather than closed it. This is the Two Generals result in practical clothing, and it is a proof, not an engineering gap.

So the ambiguity is permanent, and the design question changes shape. You stop asking "how do I find out what happened?" and start asking "how do I make both answers safe?" That single reframing is most of what this domain teaches.

ObservationSound conclusionUnsound conclusion
HTTP 500 from BtypicalB reached a decision and reports failureNothing was written — B may have committed then failed to finish
Connection refusedtypicalThis connection attempt did not reach a listenerThe request never executed — an earlier attempt may still be in flight
TimeoutprotocolNo response arrived in time. Nothing more.The request failed
What A can soundly conclude

The retry makes it worse before it makes it better

The natural response to a timeout is to retry, and retrying is usually correct. But a retry after an ambiguous outcome is *by construction* a possible duplicate — you are re-sending work that may already be done. Retry and idempotence are therefore not two separate topics; the first requires the second.

This is the origin of the practical shape you meet everywhere: at-least-once delivery plus idempotent processing. Not because exactly-once is unfashionable, but because the ambiguity above means at-most-once and at-least-once are the only two delivery behaviours a network can actually offer, and only one of them keeps working when messages are lost.

Key points

  • A timeout is the absence of a result, not a negative result.
  • Three of the five realities behind a timeout involve the work having completed.
  • No acknowledgement protocol removes the ambiguity; it relocates it.
  • Therefore: design so that both outcomes are safe, rather than trying to learn which occurred.
  • A retry after a timeout is a possible duplicate by construction — that is why retries require idempotence.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • • A sets a deadline before sending — a bound on its own waiting, not on B’s execution.
  • • B may receive, execute, commit and reply at any point relative to that deadline.
  • • The deadline expires. A stops waiting; nothing about B changes as a result.
  • • B may still be executing, and may still commit *after* A has given up and moved on.
  • • A must now choose an action that is correct whether or not B committed.
What can fail at the boundary
  • • The request is lost in the network and never arrives.
  • • The request arrives; B crashes before doing the work.
  • • B does the work and commits; B crashes before replying.
  • • B does the work and replies; the response is lost.
  • • B is simply slower than A’s deadline and is still working — the request is live.
How it fails — what an operator sees
  • • Double charge: A treats the timeout as failure and retries a non-idempotent operation. The operator sees two ledger entries with different request ids and no error in either service’s logs.
  • • Silent loss: A treats the timeout as success and does not retry. The operator sees an order stuck in pending forever, with no error anywhere.
  • • Zombie write: A gives up, marks the request failed and reverses it, then B’s original execution commits afterwards. The operator sees state that matches neither service’s view.
  • • Retry amplification: every caller in a tier retries simultaneously against an already-slow B, and A’s timeouts become the cause of B’s slowness rather than a symptom.
Where coordination is required
  • • None is required to *observe* a timeout — and that is the point. The ambiguity is local and unavoidable.
  • • Removing the ambiguity for a business effect requires shared state that both sides can consult: a deduplication record keyed by a request identifier the client chose.
  • • That shared record is itself a coordination point, with its own availability and its own failure modes — which is why "just make it idempotent" is a design decision, not a free fix.
What still holds under failure
  • • Durability of anything B already committed is unaffected — B’s commit does not depend on A hearing about it.
  • • A’s view and B’s view of the same request are now divergent, and stay divergent until something reconciles them.
  • • Any invariant that spans both services ("charged if and only if the order is paid") is temporarily unenforced.
How it recovers
  • • Detect: reconcile A’s request log against B’s effect log on a schedule; the ambiguous window is where they disagree.
  • • Contain: make the retry safe before making it fast — an idempotency key turns an unknown outcome into a repeatable one.
  • • Recover: re-drive the operation with the same key, so a duplicate collapses into the original result.
  • • Reconcile: for effects that cannot be made idempotent, compensate explicitly — a refund is a new action, not an undo.
  • • Verify: alert on the reconciliation gap itself, not only on error rates. This class of bug produces no errors.
How you would know
  • • Count of requests that ended in timeout, separately from requests that ended in an error response — most dashboards wrongly merge them.
  • • Duplicate rate at the receiver, keyed by client request id. A non-zero rate is normal; a rising one is a symptom.
  • • The reconciliation delta between caller-side and callee-side records of the same operation.
  • • Ratio of retry traffic to original traffic on the dependency — the early warning for amplification.
When it helps
  • • Always: this is a property of remote calls, not a design you opt into.
  • • The reasoning is most valuable where an operation has an external side effect — money, email, provisioning — because those are where a duplicate is expensive and a silent loss is worse.
When it hurts
  • • Reasoning about ambiguity is wasted effort for a genuinely idempotent read where a duplicate costs a little load and nothing else.
  • • Over-applying it — routing every internal call through a deduplication store — buys real latency and a new shared dependency to protect operations that could not be harmed by a duplicate.
Simpler alternatives
  • • Make the operation naturally idempotent so the question stops mattering: PUT a state rather than POST an increment.
  • • Have the client generate the identity of the effect up front, so a duplicate is detectable without a separate dedup store.
  • • Move the work behind a durable queue and let the consumer own the effect exactly once *per consumer offset* — this narrows the ambiguity rather than removing it.
  • • Accept the duplicate and reconcile afterwards, where the business can absorb it more cheaply than the coordination would cost.

Five realities, one identical observation

A timeout tells you nothing about whether the work happened
Pick which of the five realities occurred. The panel on the right is what your process actually has — and it is the same in every case.
Which reality occurred?
What the caller has
POST /charge  order_id=A-7741   deadline=2000ms
sent at        14:02:11.108
bytes received 0
status         —
error          DeadlineExceeded after 2000 ms
Identical in all five worlds. Three of the five left durable effects behind; two did not. Nothing in this evidence separates them, which is why catch (e) { markFailed(order) } is wrong here: it merges a definite negative with a total unknown.
What really happened
Hidden, which is the point: from inside the caller this panel does not exist. Decide what to do before you reveal it — whatever you choose must be correct in all five worlds at once, because you will never find out which one you were in.
safe conclusion
none about execution
the one safe design
record it as unknown and resolve it later; do not guess
Reality crashed-after: what really happened, which the caller cannot see.assumption
CallerCallee is down over this spanCalleerequest: deliveredrequestcharge(A-7741), deadline 2s (write) at t=0charge(A-7741), deadline 2scharge committed — durable (write) at t=3charge committed — durableprocess dies (crash) at t=4process diesdeadline expires — the caller learns nothing (decide) at t=6deadline expires — the caller learns nothingt=0time →t=6
delivereddelayed (dashed, long)duplicated (×2)dropped — stops short, never arriveswritecrashdecide
The caller’s lane is the same picture in all five realities: one arrow out, nothing back before the deadline. Everything that distinguishes this diagram from the other four happens where the caller cannot see it.
The request never arrivedIt crashed before committingIt committed, then crashedThe reply was lostIt is still working
What the caller observedprotocolDeadlineExceeded after 2000 ms · 0 bytes receivedDeadlineExceeded after 2000 ms · 0 bytes receivedDeadlineExceeded after 2000 ms · 0 bytes receivedDeadlineExceeded after 2000 ms · 0 bytes receivedDeadlineExceeded after 2000 ms · 0 bytes received
Executions at the callee00111
Durable effectsnonoyesyesyes
Retry, no idempotency keytypicalCharges once — the retry is the only execution.Charges once.Charges twice. The customer is billed for one order, twice.Charges twice.Charges twice, and the two attempts may interleave.
Retry with an idempotency keytypicalCharges once.Charges once.Returns the stored result of the first charge. Billed once.Returns the stored result. Billed once.The second attempt sees the key in flight: it waits or is rejected as a duplicate. Billed once.
Five realities, one row that never changes.
The timeout is not a defence against slowness; it is the thing that manufactures the ambiguity. The latency tail is unbounded, so you must impose a deadline, and the moment you do you have created a third outcome that carries no information. A system with no timeouts does not avoid this — it converts the tail into permanently held threads and connections instead. The engineering response is not a better guess: it is a third state (unknown), an operation that is safe to repeat, and a process that resolves the unknowns later.
assumptionUnder an asynchronous network — no bound on message delay — these five worlds are indistinguishable at the caller by construction, so no amount of instrumentation separates them. The idempotency-key rows assume a key scoped to this operation and stored atomically with its effects.

What people believe, and what is true

Claim

A timeout means the request failed.

Reality

It means no answer arrived. In a service with a high success rate, the work most likely *did* happen.

Claim

Raising the timeout removes the ambiguity.

Reality

It changes how often you meet it. A longer deadline still expires, and meanwhile it holds a connection and a thread open, which is how one slow dependency saturates its caller.

Claim

Retrying is safe because the first attempt failed.

Reality

The first attempt has an unknown outcome. A retry is a possible duplicate by construction.

Claim

TCP guarantees delivery, so the request arrived.

Reality

TCP guarantees ordered delivery *while the connection is healthy*; it cannot tell you the peer application processed or committed anything, and a reset discards whatever was in flight.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

A timeout says the answer did not arrive. It does not say the work did not happen. Design so both outcomes are safe.

Practical

Split your telemetry so timeouts are not counted as errors — they are a different event with different correct responses. Then make every retried operation carry a client-chosen identity, so a duplicate is recognisable at the receiver.

Advanced

The impossibility is the Two Generals problem: no finite exchange of messages over a lossy channel lets both sides reach common knowledge of a decision. Practical systems do not solve this; they choose which side absorbs the doubt, and make that side’s duplicate harmless.

Apply it

Interview questions
  • 💬 Service A calls Service B and times out. Did B execute the request?
  • 💬 You cannot make the operation idempotent — it calls a third-party API that charges a card. What now?
  • 💬 Your dashboard shows a 0.1% timeout rate on the payments dependency. What do you need to check before saying it is fine?
← Partial Failure: The Founding ConditionFailure Models: What You Are Allowed to Assume →