The question this answers
Why is "some of it is broken" so much harder to handle than "all of it is broken"?
None is added. Partial failure removes the guarantee a single machine gave you for free: that components sharing a process share a fate. After a partial failure the system is in a state that no single component can observe, and possibly one that no component considers possible.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows which of its own operations succeeded locally. It knows which peers have recently sent it something. It does not know which peers are alive, which of its own outbound messages were acted on, or what the system as a whole currently looks like — and it never will, because by the time any answer arrives it describes the past.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Failing as a unit is a feature
On one machine, a crash is clean. Every component dies together, nothing is left holding half of a decision, and the recovery story is "start it again and read the log". Ugly, but *comprehensible*: there is exactly one thing that happened and one place to look. Shared fate is a guarantee, and it is worth naming as one because distribution takes it away deliberately.
With components on separate machines, failures become independent, which is the thing you wanted and also the thing that hurts. Service B can be down while A is fine. B can be up but unreachable from A and reachable from C. B can be up and reachable and returning correct answers for 90% of requests. B can be halfway through a batch of ten items with three committed. A can crash after B committed but before A recorded that it had. Each of these leaves the system in a state where different components hold contradictory and individually-correct beliefs.
That is the founding condition, and it explains why the rest of this domain looks the way it does. Idempotency exists because a partial failure leaves an unknown outcome that must be re-driven. Sagas exist because a partial failure in the middle of a multi-step operation leaves steps that must be undone by hand. Reconciliation exists because partial failures leave divergence that nothing else will notice.
The three shapes it takes
Partial in space. Some components are affected and others are not. This is the shape people expect, and it is the least troublesome because it is at least visible: something is erroring, somewhere. The subtlety is that "affected" is relative to the observer, since a partition affects the *pair*, not the node.
Partial in time. An operation is halfway done. Three of five items were written, the first two steps of a saga committed, half the messages in a batch were published. The system is in an intermediate state that the design may never have named, and which is now externally visible. API Design has the endpoint-level treatment of reporting this; the systems question is whether the intermediate state is a legitimate state with defined behaviour, or an accident.
Partial in knowledge. The work completed and the initiator does not know. This is the shape that produces silent, expensive bugs, because nothing is erroring anywhere — the only evidence is that two records disagree, and only if someone compares them. A Timeout Tells You Nothing About Whether It Happened is this shape in its purest form.
The third shape is the one worth designing around, because the first two announce themselves and the third does not.
| What the operator sees | What it needs | |
|---|---|---|
| Partial in spacetypical | Errors on some paths, healthy metrics elsewhere | Containment: bulkheads, degraded modes, blast-radius limits |
| Partial in timetypical | Records stuck in an intermediate state | Named states, resumption, compensation |
| Partial in knowledgeprotocol | Nothing. Two systems disagree and neither errored | Idempotent re-drive plus reconciliation |
Why "handle the error" is not the answer
The instinct trained by single-machine programming is that failures are exceptional, so they belong in an error path bolted to the side of the happy path. Under partial failure that instinct produces systems that are wrong in a hundred small ways, because there is no single error path — there is a different residual state for each point at which the operation could have been interrupted, and most of them are not errors from anyone’s local perspective.
The shift is to design the states, not the error handling. An order is not "created, and possibly an exception happened". It is pending_payment, paid, reserving, reserved, shipped, payment_unknown, reconciling. Each is a state the system can be in, each has a defined next action, and each can be counted on a dashboard. That transformation — from exceptions to states — is what makes a distributed system operable, and it is why API Design’s treatment of resource state machines and the transactions module here keep arriving at the same place.
The second half is that some states can only be exited by a process running *outside* the original request: a sweeper that finds payment_unknown records and asks the payment service what actually happened. That process is not a nice-to-have. Under partial failure it is the only thing that closes the loop, because the component that would have closed it is the one that failed.
1// Two outcomes assumed. Every partial failure lands in the2// same bucket, and the bucket is a lie in most of the cases.3async function placeOrder(o: Order) {4 try {5 await payments.charge(o)6 await stock.reserve(o)7 await orders.markPlaced(o)8 } catch (e) {9 await orders.markFailed(o) // <- charged? reserved? nobody knows10 }11}12 13// States, each with a defined next action and a metric.14async function placeOrder(o: Order) {15 await orders.setState(o, 'charging')16 const charge = await payments.charge(o, { key: o.id })17 if (charge.kind === 'unknown') return orders.setState(o, 'charge_unknown')18 if (charge.kind === 'rejected') return orders.setState(o, 'payment_declined')19 20 await orders.setState(o, 'reserving')21 const res = await stock.reserve(o, { key: o.id })22 if (res.kind === 'unknown') return orders.setState(o, 'reserve_unknown')23 if (res.kind === 'rejected') return orders.setState(o, 'refund_pending')24 25 await orders.setState(o, 'placed')26}27// charge_unknown / reserve_unknown / refund_pending are resolved by a28// sweeper, not by the request that created them — the request may be gone.Availability stops being a boolean
On one machine, "up" is a fact. In a distributed system it is a question with a scope: up for whom, for which operation, with what freshness? A system can be up for reads and down for writes; up in one region and down in another; up for 90% of tenants and down for the one whose shard is offline; up in the sense of returning 200s and down in the sense that the data is four hours stale.
This is why aggregate availability metrics mislead so consistently. A service at 99.95% availability might be perfectly healthy for everyone except one tenant, for whom it has been unusable all week — a fact that is invisible in the aggregate and obvious in a per-tenant breakdown. Performance owns the SLI/SLO machinery; the distributed-systems point is that the right unit of availability is the user-visible operation, not the process, and partial failure is exactly why.
It is also why graceful degradation is a design activity rather than a fallback. If a recommendation service is down, the product page should render without recommendations. That behaviour has to be built, decided and tested; the default behaviour of a partial failure is to propagate, so an unhandled dependency failure becomes a page that does not render at all.
Key points
- A single machine fails as a unit; a distributed system fails in pieces, at different times, with each piece holding a different picture.
- Three shapes: partial in space, partial in time, partial in knowledge. The third is silent and the most expensive.
- Design named states, not error paths — each residual state needs a defined next action.
- Some states can only be resolved by a process outside the original request.
- Availability stops being a boolean and becomes per-operation, per-tenant, per-freshness.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • An operation touches several components, each of which commits independently.
- • A failure interrupts the sequence at some point, leaving a prefix committed.
- • The initiator may or may not learn how far the sequence got.
- • Each component continues acting on its own local view, which is correct for it and incomplete overall.
- • The system as a whole is now in a state that requires an external process to observe and repair.
- • One component crashes mid-operation, leaving a committed prefix.
- • A network partition isolates a component that has already acted.
- • The initiator crashes after the effect but before recording it.
- • A component is slow rather than down, so the operation completes after everyone has moved on.
- • A batch is partially applied, and the caller learns only an aggregate result.
- • Orphaned effect: money is captured and no order exists. The operator sees a payment provider report that does not reconcile with the order table, and no errors on either side.
- • Stuck intermediate state: records accumulate in
reservingand never move. The operator sees a state-count gauge growing monotonically — the cheapest possible detector and the one most often missing. - • Degradation by propagation: a non-critical dependency fails and the entire page 500s. The operator sees an outage whose severity far exceeds the failed component’s importance.
- • Hidden per-tenant outage: aggregate availability is 99.95% and one tenant has been at 0% for a day. The operator sees green dashboards and an escalating support thread.
- • Partial failure is not caused by lack of coordination — it is what coordination protocols exist to manage.
- • Making a multi-component operation atomic requires distributed commit, which couples the availability of all participants: any one of them being down blocks the others.
- • The usual alternative is to accept the intermediate states and manage them, which trades atomicity for availability and moves the work to reconciliation.
- • Every component’s local state remains valid and durable according to its own rules.
- • Cross-component invariants are unenforced until something reconciles.
- • Requests that arrive during the failure produce more intermediate states, so the population needing repair grows while the failure lasts.
- • Detect: count records in each intermediate state, and alert on age rather than on volume — depth is normal, age is not.
- • Contain: stop new work from entering the affected path so the repair population stops growing.
- • Recover: re-drive each stuck operation idempotently, asking the downstream what actually happened rather than assuming.
- • Reconcile: compare the two sides’ records of the same operations and repair the delta with a documented rule.
- • Verify: confirm the intermediate-state counts return to their normal steady level, which is rarely zero.
- • A gauge per intermediate state, plus the age of the oldest record in each — a state that only grows is the signature of partial failure with no sweeper.
- • Reconciliation delta between each pair of components that share an operation.
- • Availability sliced by operation and by tenant, not aggregated — partial failure hides perfectly in an average.
- • Ratio of degraded responses to full responses, so graceful degradation is visible rather than silent.
- • This framing is essential for any operation that touches more than one component — which is nearly all of them once there is a database and a queue.
- • It pays off most in systems with external side effects, where an orphaned effect costs money or trust.
- • Modelling elaborate state machines for a read-only path with no side effects adds ceremony that will never be exercised.
- • Building reconciliation for components that already share a transaction is duplicated effort — inside one store, the database handles it.
- • Keep the whole operation inside one transactional store, so failures are total rather than partial. This is the strongest option and is available more often than people assume.
- • Make the operation a single durable message that a consumer processes at-least-once, so the intermediate state lives in the queue rather than in your tables.
- • Reorder the steps so the reversible ones happen first and the irreversible one is last — a partial failure then leaves nothing that needs undoing.
- • Accept the intermediate state and expose it honestly to users, rather than pretending the operation is atomic.
Some of it is broken: the state nobody can see
What people believe, and what is true
Good error handling covers partial failure.
Error handling covers the cases that produce errors. The expensive shape of partial failure produces no error anywhere — two components simply disagree.
A distributed transaction removes partial failure.
It converts it into blocking: a participant that fails between prepare and commit holds locks until it returns. See The Blocking Window: When 2PC Stops and Waits.
If each service is 99.9% available, the system is 99.9% available.
A chain of dependencies multiplies. Availability is a property of the operation across all of its dependencies, not of any one service.
Retrying the whole operation fixes a partial failure.
It re-executes the prefix that already committed. Without idempotence that makes things worse, which is why the two topics are inseparable.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Distributed systems fail in pieces. Some components act, others do not, and each holds a correct but incomplete picture. Design for the residual states, not for an error path.
Practical
Name every intermediate state an operation can be left in. Give each one a next action and a metric with an age. Build the sweeper that resolves them, because the request that created them may be gone. Then slice availability by operation and tenant, because a partial outage is invisible in an aggregate.
Advanced
The deep version is that partial failure makes the *global state* unobservable. There is no moment at which anyone can see the whole system, so "what state is the system in" is not a question with an answer — only a consistent cut, assembled after the fact from causally-related local snapshots, comes close. That is what A Consistent Cut, Without Stopping the World formalises, and it is why post-incident reconstruction of a distributed system is genuinely difficult rather than merely tedious: you are not retrieving a record of the global state, you are inferring one that never existed as such.
Apply it
- ⚡ Design the state machine for a checkout that touches payments, inventory and notifications, naming every state a partial failure can leave behind and the process that resolves each one.
- 💬 Why is partial failure harder to handle than total failure?
- 💬 An order is charged but not reserved because a service crashed in between. Who fixes it, and how do they find it?
- 💬 Your service is 99.95% available in aggregate. Why might that number be hiding an outage?