The question this answers
What exactly does the network do to my messages, and which of it can I stop worrying about?
Over an asynchronous network you get at most this: a message that is delivered was sent by someone claiming to be the sender, and — if you use a connection-oriented transport and it stays up — bytes within one connection arrive in order and unduplicated. Nothing about *whether* it arrives, *when*, or whether the peer application saw it.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows the messages it has received and the ones it has sent. It does not know whether an unacknowledged message is lost, in flight, or already processed. It cannot tell "the peer did not answer" from "the peer answered and the answer was lost" from "the peer is unreachable from me but reachable from everyone else".
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Five behaviours, and which layer removes which
It is worth being precise about which problems TCP solves, because engineers routinely over-credit it. Within a single healthy connection, TCP gives you ordered, deduplicated, retransmitted delivery of a byte stream. That removes reordering and duplication at the transport layer, within that connection. It does not remove loss — it converts loss into delay, and if the delay exceeds the retransmission budget it converts it into a reset. Networking owns the mechanism; what matters here is what remains.
What remains is everything that matters for correctness. Across two connections, ordering is gone. Across a reset, in-flight bytes are gone with no notification of how many the peer processed. Across a retry at the application layer, duplication is back — this time as a *semantic* duplicate that TCP was never in a position to see. And a partition is invisible to TCP by construction: a connection that cannot be established and a connection that was never attempted look the same.
| TCP within one connection | Across connections / after a reset | What you must do | |
|---|---|---|---|
| Lossprotocol | Retransmits, then gives up | Not handled | Retry at the application layer, with a deadline |
| Delayprotocol | Made worse by retransmission | Unbounded | Impose a deadline; treat expiry as unknown, not failure |
| Reorderingprotocol | Removed | Not handled | Carry a sequence number or version if order matters |
| Duplicationprotocol | Removed at byte level | Reintroduced by your own retries | Idempotent handling keyed by a caller-chosen id |
| Partitionassumption | Invisible | Invisible | Decide what each side does when it cannot reach the other |
The partition is the one that changes designs
Loss, delay, reordering and duplication are properties of individual messages and are handled with per-message machinery: retries, sequence numbers, idempotency. A partition is different in kind. It is a sustained condition in which the cluster splits into groups where messages flow within a group and not between groups — and, critically, every node still works. Nothing has crashed. Each side sees a subset of the cluster go silent and has to decide what that means.
The two available answers are the whole of the CAP argument: keep serving on both sides and accept that the two sides diverge, or refuse to serve on the side that cannot establish a majority and accept the unavailability. There is no third option that preserves both, and no amount of engineering budget buys one. Architecture owns the pattern-level treatment; the consistency module here does the precise version.
Partitions are also rarer and weirder than the textbook picture. A partition is often *asymmetric* — A can send to B but B cannot send to A — or *partial*, affecting one port, one protocol, or traffic above a certain size. Those are worse than a clean split because they defeat the intuition that both sides observe the same silence.
- Node 1 — majority side
- n1believes “nodes 4 and 5 have failed”✕ and it is false
- n4believes “nodes 1, 2 and 3 have failed”✕ and it is false
- n4believes “node 1 is still the leader”✓ and it is true
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
The fallacies, restated as design questions
The classic list — the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change, there is one administrator, transport cost is zero, the network is homogeneous — is usually presented as trivia. It is more useful as a checklist of assumptions to hunt for in your own code, because each one shows up as a specific line someone wrote.
"The network is reliable" shows up as a call with no retry. "Latency is zero" shows up as a loop that issues one request per row. "Topology does not change" shows up as a cached IP address, or a connection pool that never notices its peers were replaced. "The network is secure" shows up as a service that trusts a header because it came from inside the VPC — which is Security’s subject, and worth reading, because the machine boundary is also a trust boundary.
The one worth internalising above the others is bandwidth is not infinite and transport is not free. A design that moves a gigabyte between services per request is not slow because of code; it is slow because of physics and it will also be expensive, particularly across zones or regions where the bytes are metered.
- Reliable → where is the retry, and is the operation safe to repeat?
- Zero latency → how many round trips does one user action cost?
- Infinite bandwidth → what is the largest payload this endpoint can be asked for?
- Secure → what does this service trust, and why does it believe the caller?
- Stable topology → what happens when every peer address changes at once?
- Free transport → which of these bytes cross a priced boundary?
What you do about it
The response is not a library. It is three habits. First, name the deadline for every crossing, so an unbounded delay becomes a bounded one you decided on. Second, make the repeat safe, because a deadline plus a retry is the only way to survive loss, and a retry is a duplicate by construction. Third, decide the partition behaviour explicitly — for each piece of state, whether the minority side may serve reads, serve writes, or must refuse.
Everything else in this domain is a refinement of those three. The idempotency module does the second in detail; the consistency and consensus modules do the third; the deadlines module does the first across a call graph rather than a single hop.
Key points
- The network may lose, delay, reorder, duplicate, or partition. All five are normal operating conditions.
- TCP removes reordering and duplication within one connection, converts loss into delay, and is blind to partitions.
- A partition is not a crash: every node is working, and each side sees the other as gone.
- Asymmetric and partial partitions are more common and more confusing than the clean textbook split.
- Deadline, safe repeat, explicit partition behaviour — those three cover most of the practice.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A message is handed to the transport, which may buffer, fragment and retransmit it.
- • Queues at each hop add delay proportional to load; under saturation, delay is unbounded rather than merely large.
- • A lost packet is retransmitted after a timeout; repeated loss escalates to a connection reset that discards in-flight state.
- • Application-level retry re-sends the request over a new connection, with no relationship to the previous attempt unless you created one.
- • A link or routing failure isolates a group of nodes; from inside each group, the other group is simply silent.
- • A packet is dropped by a saturated queue rather than a broken link — indistinguishable from the outside.
- • The connection is reset mid-request and the caller cannot tell how many bytes the peer application consumed.
- • A middlebox silently drops idle connections, so the first request after a quiet period fails and the second succeeds.
- • MTU or payload-size limits fail only for large messages, producing a bug that correlates with data volume rather than with code paths.
- • Traffic is partitioned in one direction only, so one side sees a healthy peer and the other sees silence.
- • Grey failure: the link is not down, it is losing 2% of packets. The operator sees normal availability metrics, a p99 that has tripled, and every service blaming the next one.
- • One-way partition: A reaches B, B cannot reach A. The operator sees B logging successful request handling while A logs timeouts for the same request ids.
- • Idle-connection reset: the first request after a quiet period always fails. The operator sees an error rate that correlates with traffic *troughs*, not peaks.
- • Size-correlated failure: requests above a threshold fail while small ones succeed. The operator sees an error rate that tracks average payload size and no pattern in endpoint or tenant.
- • Cross-zone cost blowout: a chatty design works fine functionally, and the operator sees it first on the bill rather than on a dashboard.
- • None of these behaviours require coordination to *observe* — each node sees its own local evidence.
- • Agreeing on what the evidence means (is node 4 down, or partitioned?) requires a majority, and that is exactly what a partition can prevent on one side.
- • This is the structural reason a minority partition cannot safely keep making authoritative decisions: it cannot reach the quorum that would tell it whether it is the majority.
- • Messages already delivered and processed remain processed; the network cannot un-deliver.
- • During a partition, each side remains internally consistent and the two sides diverge from each other.
- • Any invariant defined across the partition boundary is unenforced for the duration, whether or not the system reports an error.
- • Detect: measure reachability pairwise between nodes rather than centrally, because a central prober has its own single view.
- • Contain: prevent the minority side from taking authoritative actions, using leases and fencing rather than good intentions.
- • Recover: on heal, expect both sides to have progressed; plan the merge before you need it.
- • Reconcile: run anti-entropy across the formerly-partitioned sides — see Anti-Entropy: Repairing Divergence Nobody Reported and Merkle Trees: Finding the Difference Without Reading the Data.
- • Verify: check that the reconciled state satisfies the invariant, not merely that replication caught up.
- • Pairwise connectivity matrix between nodes; a partition shows as a block structure that a per-node health check cannot reveal.
- • Retransmit rate and packet loss per link, separately from application error rate — grey failure lives here.
- • Distribution of request sizes among failures versus successes, which is what surfaces MTU and limit problems.
- • Cross-zone and cross-region byte volume per request path, because transport cost is a design signal, not just a finance one.
- • Always, but it earns its keep most when designing anything that fans out across zones, or any protocol where one side holds exclusive rights.
- • Defending against partitions inside a single process, or between two containers on the same host that share fate anyway, adds machinery you will never exercise.
- • Treating every internal call as potentially partitioned, when the real risk is that the whole zone goes at once and both sides die together.
- • Move the interaction off the request path onto a durable log, so loss and reordering become the log’s problem and the consumer’s offset becomes the ordering.
- • Collapse the boundary: two components that must never disagree are cheaper as one deployable unit than as two with a consensus protocol between them.
- • Accept divergence by design — choose a data type that merges without coordination, so a partition costs staleness rather than correctness. See CRDTs: Deterministic Merge, Not Correct Merge.
Loss, delay, reordering, duplication, partition — and which layer removes which
| Behaviour | TCP, within one connection | Still visible here? | What you must do |
|---|---|---|---|
| Loss | Retransmits, then gives up | no | Application retry with a deadline |
| Delay | Made worse by retransmission | yes | Impose a deadline; treat expiry as unknown, not failure |
| Reordering | Removed within one connection | yes | Carry a sequence number or version if order matters |
| Duplication | Removed at byte level | no | Idempotent handling keyed by a caller-chosen id |
| Partition | Invisible | no | Decide what each side does when it cannot reach the other |
What people believe, and what is true
TCP guarantees my message arrives.
It guarantees ordering and retransmission while the connection is healthy. A reset discards in-flight data and tells you nothing about how much the peer consumed.
Partitions are rare enough to ignore.
Full clean partitions are rare; grey failures, asymmetric reachability and single-link loss are routine, and they exercise the same code paths with less obvious symptoms.
Both sides of a partition see the same thing.
Only under a symmetric split. Asymmetric partitions are common and produce logs that appear to contradict each other.
A retry makes the system reliable.
A retry makes delivery more likely and duplication certain. Reliability is retry plus a safe repeat, never retry alone.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Networks lose, delay, reorder, duplicate and partition. TCP handles two of those inside one connection and nothing across connections. Everything else is your design.
Practical
For each remote interaction, write down four answers: the deadline, the retry policy, what makes the repeat safe, and what the caller does when the callee is unreachable for minutes rather than milliseconds. If any answer is missing, the behaviour still exists — it is just accidental.
Advanced
The uncomfortable case is the partial partition: node A reaches B and C, B reaches C but not A. No node has a complete picture, majorities can be unstable, and leader elections can thrash indefinitely because the set of mutually-reachable nodes keeps changing. Systems that handle this well do so by requiring a candidate to demonstrate reachability to a majority *before* disrupting a functioning leader — a pre-vote phase — rather than by detecting the topology.
Apply it
- ⚡ A service starts failing only for requests carrying a large attachment list. Walk through the network-level hypotheses before touching application code.
- ⚡ Two services each log that the other is unavailable, but only one of them shows incoming requests. Explain what topology produces this.
- 💬 Which of loss, delay, reordering, duplication and partition does TCP remove, and under exactly what conditions?
- 💬 Describe a one-way partition and what the two sides’ logs would look like.
- 💬 Your error rate correlates with traffic troughs rather than peaks. What is your first hypothesis?