The question this answers
The coordinator crashed after everyone voted yes. Why can the participants not simply decide for themselves?
Safety survives every failure: participants never disagree about the outcome, and no participant commits unless all voted yes. Liveness does not survive coordinator failure: a prepared participant may block for an unbounded time, and the only bound available is the coordinator’s recovery time — which is a human number unless the coordinator is replicated.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A prepared participant knows: it voted YES, it is durably able to commit, and no decision has arrived. It cannot infer the decision from silence, because silence is produced both by "the coordinator decided ABORT and the message was lost" and by "the coordinator decided COMMIT and crashed mid-broadcast" — and by "the coordinator is fine and slow". This is A Timeout Tells You Nothing About Whether It Happened applied to the one message that matters most in the entire system.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The uncertainty window, drawn exactly
There is a precise interval in which a participant is *uncertain*: it begins when the participant forces its YES vote to disk, and it ends when the decision arrives. Inside that interval the participant cannot abort, because another participant may already have committed; and it cannot commit, because another participant may have voted NO. It has no legal move. It waits.
Everywhere else in the protocol, a timeout has a safe default. Before voting, a participant may abort unilaterally — nobody has committed. The coordinator may abort while collecting votes. Only in the uncertainty window is there no safe default, and that is not an implementation oversight: it is a theorem. Any atomic commit protocol has a window in which a participant’s decision depends on information held elsewhere, because a commit requires knowing something about the other participants that cannot be derived locally.
What makes it operationally severe is what the participant is holding while it waits. Locks, obviously — but in an MVCC database, also the transaction horizon. A prepared transaction in PostgreSQL pins the oldest visible transaction id, so vacuum cannot remove dead tuples anywhere in that database, not merely in the tables the transaction touched. A single forgotten in-doubt transaction can bloat an entire cluster over a weekend.
Cooperative termination: partial relief, not a fix
A blocked participant is not required to sit in silence. It can ask its peers, and this is the *cooperative termination protocol*. If any peer has already received the decision, it can share it. If any peer has not yet voted, it can abort and tell everyone the transaction is doomed. Real XA transaction managers do a version of this during recovery.
It helps in exactly the cases where the coordinator crashed *during* the broadcast, which is a common shape. It does not help when the coordinator crashed *before* the broadcast, because then no participant has the information and the set of uncertain nodes is complete. The protocol is not blocking-free; it merely narrows the window.
And it introduces its own hazard: participants must now know each other, which is an extra coupling, and a participant that answers "I do not know" contributes nothing while consuming a round trip. Under a network partition, every peer answers "I do not know" and you have added latency to a stall.
Heuristic decisions — where atomicity actually breaks
Every real XA implementation includes an escape hatch: an operator may force a prepared transaction to commit or roll back. JTA calls the outcome HeuristicCommit, HeuristicRollback or HeuristicMixed; Postgres exposes it as COMMIT PREPARED / ROLLBACK PREPARED on a bare gid. It exists because a business cannot leave a table locked for three days while waiting for a coordinator to be rebuilt.
It is also the one mechanism that can break 2PC’s safety guarantee, and it does so silently. An operator who rolls back a prepared transaction on participant A while participant B has already committed has produced exactly the mixed outcome the protocol was bought to prevent — and there is no automated detection for it, because both participants believe they behaved correctly. HeuristicMixed is the standard’s name for "we know the outcome disagreed", and by the time you see it, the damage is in your data.
The practical lesson is not "never use heuristics". It is that the escape hatch must be operated with a written record of which transaction was forced and how, so that reconciliation has something to work from. A heuristic decision is a decision to move the problem from the transaction layer to the business layer, which is exactly what a compensation is.
# Incident 4821 — coordinator host lost, disk unrecoverable # 6 in-doubt transactions, all older than 5h, blocking vacuum on orders. gid participant forced at by xa-7f1c9e04-orders-482913 orders-db COMMIT 2026-08-24T09:02:11Z dbre xa-7f1c9e04-payments-482913 payments-db COMMIT 2026-08-24T09:02:40Z dbre xa-91ba22d1-orders-482980 orders-db ROLLBACK 2026-08-24T09:04:02Z dbre xa-91ba22d1-payments-482980 payments-db ROLLBACK 2026-08-24T09:04:19Z dbre ... # Each pair MUST be forced the same way. Cross-check before running. # Reconciliation job R-4821 scheduled to diff both sides for the window.
The real fix: make the coordinator not a single point
The blocking window exists because exactly one node holds the decision. That is a availability problem with a well-known solution: replicate the decision. Paxos Commit replaces the coordinator’s local log with a consensus group, so the decision survives the loss of any minority of coordinator replicas and a new coordinator can read it. The uncertainty window is then bounded by leader election — seconds — rather than by hardware replacement.
Google Spanner takes the same idea further and makes each *participant* a Paxos group as well. Now neither participant failure nor coordinator failure blocks anything: a failed replica is replaced by its peers, and 2PC runs between highly available groups rather than between single machines. This is why Spanner can offer cross-shard, cross-region transactions with strict serializability and still be described as running two-phase commit. The protocol was never the problem; the single-node coordinator was.
This reframes the honest advice. If you need atomic commit across resources you control, do not ask "is 2PC bad?" — ask "is my coordinator a single process with a local log file?". If it is, your uncertainty window is bounded by an operator’s pager. If it is replicated, the classic objection largely evaporates and the remaining cost is the round trips and the lock hold time, which are ordinary engineering trade-offs.
| Coordinator design | Window bounded by | Typical duration | Residual risk |
|---|---|---|---|
| Single process, local logtypical | Process restart | Seconds to minutes | Log survives; nothing else does |
| Single host, durable disktypical | Host recovery | Minutes to hours | Disk loss ⇒ heuristic decisions |
| Consensus-replicated (Paxos Commit)protocol | Leader election | Sub-second to seconds | Minority loss tolerated |
| Replicated participants too (Spanner-style)protocol | Leader election | Sub-second to seconds | Cost: more round trips, higher latency |
Being fair to 2PC
It is easy to over-learn the blocking story and conclude that 2PC should never be used. That conclusion does not survive contact with the systems you rely on. Multi-shard commits in CockroachDB, TiDB, YugabyteDB and Spanner are two-phase. A database enlisting with a message broker in the same rack is two-phase. Cross-partition writes inside a single logical database are, essentially always, two-phase.
The failure mode has a shape, and you should judge the fit against that shape: how long are locks held, how likely is the coordinator to vanish, how bad is a stall, and can you replace the coordinator with a replicated one? A ten-millisecond multi-shard commit inside one cluster fails that test on none of those axes. A cross-company XA transaction that stays open while a user fills in a form fails it on all four.
The saga is not a strictly better protocol; it is a different trade. It never blocks, and in exchange it never offers isolation, and every step needs a compensation that is itself a distributed operation that can fail. Choosing it is choosing to move correctness work from the protocol into your business logic — sometimes right, never free.
Key points
- The uncertainty window runs from the participant forcing its YES vote to the arrival of the decision; inside it there is no safe unilateral action.
- Blocking is a theorem about atomic commit, not a bug in 2PC — a commit decision cannot be derived from local state alone.
- A prepared transaction holds locks and, under MVCC, pins the vacuum horizon for the whole database.
- Cooperative termination narrows the window but cannot close it when the coordinator crashed before broadcasting.
- Heuristic decisions are the escape hatch and the one way atomicity actually breaks — silently, and only visible later in the data.
- Replicating the coordinator through consensus bounds the window by leader election; replicating the participants too is what Spanner does.
- 2PC inside one trust domain with short transactions and a replicated coordinator is a good, widely used design.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Participants force their YES votes and enter the prepared state, opening the uncertainty window.
- • The coordinator logs its decision. Whether it has broadcast yet is invisible to participants.
- • The coordinator fails. Participants time out waiting for a decision that they may not infer.
- • Participants optionally query each other; if any received the decision, it propagates. Otherwise all remain uncertain.
- • Participants continue to hold locks and, in MVCC systems, hold back the transaction horizon.
- • On coordinator recovery, it reads its log and re-broadcasts the decision; participants apply it and release resources.
- • If the coordinator’s log is lost, an operator forces each participant — a heuristic decision, made consistently by hand or not at all.
- • The coordinator crashes after logging COMMIT and before sending it — the canonical blocking case.
- • The coordinator crashes after sending to some participants and not others — partial knowledge, resolvable by cooperative termination.
- • A network partition isolates the coordinator from a subset of participants, which is indistinguishable from a crash to them (Crashed or Just Slow: The Distinction You Cannot Make).
- • The coordinator’s durable log is on a disk that dies, so the decision is genuinely lost and no correct resolution exists.
- • A participant crashes while prepared and, on restart, must discover it has an in-doubt transaction and go looking for the answer.
- • An operator forces different decisions on different participants, producing a mixed outcome that violates atomicity.
- • Silent stall with green dashboards: application requests to unrelated features start timing out because a participant’s connection pool is full of sessions waiting on locks held by an in-doubt transaction. Nothing is logging errors; the queue is the symptom.
- • Weekend bloat: an in-doubt transaction on Friday evening blocks autovacuum for 60 hours. Monday’s symptom is slow queries and a disk usage alert; the cause is a transaction nobody remembers starting.
- • Recovery scan reveals in-doubt work: after a coordinator restart,
XA RECOVERorpg_prepared_xactslists transactions the application has long since reported to users as failed. The users were told one thing; the data says another. - • HeuristicMixed in the transaction manager log: the outcome disagreed across participants. The operator sees this hours after the fact, and the only remedy is a business-level reconciliation of every row in the window.
- • Cascading timeouts upstream: callers of the blocked service exhaust their own deadlines and retry, so a stall on one participant becomes load amplification across the tier (One Retry per Tier Is Not One Retry — It Multiplies).
- • The uncertainty window is coordination made visible: the participant is waiting for information it cannot generate, and its availability is now coupled to a machine it does not control.
- • Cooperative termination replaces coordinator dependence with peer dependence — still coordination, with a different failure surface.
- • Consensus-replicating the decision is real coordination with a real price: one extra round trip on the commit path in exchange for a bounded blocking window.
- • The trade is availability of the *decision* against latency of the *transaction*, and it is the same trade Coordination Couples Availability describes everywhere in this domain.
- • Atomicity holds — every automatic outcome is unanimous, no matter what fails.
- • Availability of the locked resources does not hold, and the loss extends to work that has nothing to do with the transaction.
- • Durability holds: whatever any participant committed stays committed, and prepared state survives restart precisely so recovery is possible.
- • Isolation holds in the sense that nobody sees the half-committed state — because nobody can read the locked rows at all, which is the availability cost restated.
- • Detect: alert on prepared-transaction count and maximum age per participant. Treat any age beyond a few seconds as a page, not a warning.
- • Contain: kill nothing yet — first cap the damage by reducing new traffic to the affected participant and by capping concurrent prepared transactions.
- • Recover: bring the coordinator back and let it re-broadcast from its log. This is the only resolution that preserves the guarantee, and it should be tried before any manual action.
- • Reconcile: if and only if the coordinator log is unrecoverable, force each participant of a given global transaction the *same* way, record what was forced, and schedule a business-level diff of both sides for the affected window.
- • Verify: after recovery, confirm the prepared-transaction count is zero, that vacuum has caught up, and that reconciliation found no cross-participant disagreement.
- • Prepared transactions: count, maximum age, and age histogram, per participant. The single highest-value 2PC signal.
- • Autovacuum lag and oldest-xmin age, which reveal an in-doubt transaction long before anyone reads a transaction table.
- • Coordinator liveness and its log flush latency, tracked as a dependency of every participant rather than as its own service.
- • Lock wait time and connection pool saturation on participants — the first place a stall becomes user-visible.
- • Heuristic decision counter from the transaction manager. It should be zero, and any non-zero value is an incident record.
- • Knowing the window’s exact boundaries is what lets you bound it: it tells you the metric to collect and the coordinator property to fix.
- • Where the coordinator can be made fault-tolerant, this analysis converts 2PC from "avoid" to "measure the lock hold time and decide".
- • In an incident, recognising the shape immediately — locks held, no errors, coordinator unreachable — cuts diagnosis from hours to minutes.
- • Concluding "2PC is bad" and switching to a saga for a ten-millisecond intra-cluster transaction, trading a bounded stall risk for a permanent obligation to write and operate compensations.
- • Relying on heuristic decisions as routine practice, which converts a strong guarantee into an unrecorded manual process.
- • Adding cooperative termination between participants that live in different failure domains, so a partition makes every participant spend a round trip learning nothing.
- • Replicate the coordinator with consensus so the decision survives its failure — the direct fix for the actual problem.
- • Use a saga and accept visible intermediate state instead of a blocking window (Sagas: Trading Isolation for Availability).
- • Shorten the window by design: fewer participants, faster prepares, participants co-located, transactions that touch fewer hot rows.
- • Remove the need for atomic commit by relocating data so the invariant is local (Exactly One Component Owns Each Piece of State).
- • Set an aggressive prepared-transaction timeout that automatically aborts, accepting that this can violate atomicity, only where the business impact of a mixed outcome is provably smaller than the impact of a stall — and reconcile afterwards.
The blocking window, priced
| Knows | Does not know | May do | |
|---|---|---|---|
| Prepared participantprotocol | It voted yes and its undo/redo is durable | Whether anyone else voted yes; whether a decision exists | Wait. Ask other participants. Nothing else. |
| Coordinator (after recovery)protocol | Its own forced log record, if it wrote one | Which participants heard the broadcast | Re-broadcast the logged decision; abort if none was logged |
| Operatortypical | That queries are slow and the pool is full | That the cause is one in-doubt transaction | Force a heuristic decision — and own the mixed outcome |
What people believe, and what is true
A blocked participant should just time out and abort.
Another participant may already have committed. Unilateral abort in the uncertainty window is exactly the action the prepared state exists to forbid, and taking it is how mixed outcomes are created.
2PC is always a bad idea because of the blocking window.
Inside one trust domain with short transactions and a replicated coordinator, the window is bounded by leader election and the protocol is used successfully at enormous scale — including inside databases you already run.
Three-phase commit solves this.
Only under synchronous timing assumptions your network does not satisfy. Under partitions 3PC can produce inconsistent outcomes, which is a strictly worse failure than blocking.
The decision is lost when the coordinator crashes.
The decision is durable in the coordinator’s log — that is why it is forced before broadcast. It is *unreachable*, not lost. The difference is what makes recovery correct rather than heuristic.
Sagas avoid this problem entirely, so they are safer.
They avoid blocking by giving up isolation. Their equivalent stall is a saga stuck in COMPENSATING because a compensating action keeps failing — and that stall has no protocol-level resolution at all.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Between voting yes and hearing the decision, a participant has no legal move. If the coordinator dies in that gap, it waits — holding locks — until the coordinator comes back.
Practical
Monitor prepared transactions by age on every participant and page on them. Keep the coordinator’s log as durable as your data. Never force a heuristic decision on one participant without forcing every participant of that global transaction the same way, and record what you did.
Advanced
Cooperative termination resolves the case where some participant learned the decision, which covers crashes during broadcast but not before it. The general result is that no atomic commit protocol is non-blocking in an asynchronous system with crash failures — the same impossibility that limits consensus. The engineering response is to make the decision-holder highly available rather than to look for a non-blocking protocol.
Internals
Paxos Commit replaces the coordinator’s single log with 2F+1 acceptors and has each participant’s vote go into a Paxos instance; the transaction commits when every participant’s instance has chosen YES. It costs one more message delay than 2PC and tolerates F failures. Spanner composes this with replicated participants: each shard is a Paxos group, the coordinator is the leader of one of the participating groups, and the prepare record is replicated through that group’s log before the vote is sent — so a coordinator crash is a leader election, and the new leader reads the decision from the replicated log.
Apply it
- 🔧 Simulate a coordinator crash between the decision log write and the broadcast. Show that the participants block, then implement cooperative termination and show which crash timings it rescues and which it does not.
- 🔧 Leave a prepared transaction open in a test PostgreSQL instance for an hour under write load, and measure the table bloat and the oldest-xmin age.
- ⚡ At 02:00 a participant database becomes unresponsive to all queries on one table. CPU is low, no errors are logged, and the coordinator host is unreachable. Diagnose it.
- ⚡ An operator forced a heuristic commit on one participant and a heuristic rollback on another for the same global transaction. Describe how you would find and repair the resulting damage.
- 💬 Define the uncertainty window in 2PC and explain why a participant inside it cannot abort.
- 💬 The coordinator’s disk is unrecoverable and six transactions are in doubt. What do you do, and what do you write down?
- 💬 How does Spanner run two-phase commit without the classic blocking objection?
- 💬 Why does an in-doubt transaction in PostgreSQL slow down queries on tables it never touched?