Datadistributed transactiontwo-phase commit2PCatomicitysaga

Distributed Transactions

Once an order, a payment and an inventory reservation live in three services with three databases, no single COMMIT covers them; two-phase commit can make it look like one but is avoided for good reasons, so production systems use sagas, compensation and the outbox pattern instead.

▶ InteractiveInterview questionDebug it
Progress
What problem does this solve?

Placing an order means writing an order, reserving stock and charging a card, and the business needs all three or none. Inside one database that is one transaction. Across three services it is three transactions that can each succeed or fail independently — this lesson is about what you lose, why the obvious fix (2PC) is rarely used, and what replaces it.

What ACID gave you, and what a service boundary takes away

In a monolith with one database, "place order" is BEGIN; INSERT order; UPDATE stock; INSERT payment; COMMIT. Transactions and ACID gives atomicity (all or nothing), isolation (nobody sees the half-done state), and a single point where the outcome is decided. A crash anywhere rolls the whole thing back; a concurrent order sees either the full result or none of it. The application code does not think about partial failure at all — that is the most underrated benefit of the Monolithic Architecture.

Split the same flow into Order, Inventory and Payment services with their own databases — the Microservices rule of database ownership — and every one of those guarantees evaporates. The order service writes its row and commits; the call to inventory times out; the payment call succeeds. Now the system holds a paid order with no stock reserved, no transaction to roll back, and no isolation: another customer can already see the stock as available. Every step is durable on its own and none of them knows about the others. This is not a bug to fix; it is the state of the world once state is partitioned across services, and the architecture must handle it explicitly.

Three services, three commits, no shared transaction
COMMIT ✓reserve — timeout?COMMIT ?chargeCOMMIT ✓Order serviceOrder DBInventory servicePayment serviceInventory DBPayment DB
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

Two-phase commit, and why it is avoided

2PC restores atomicity with a coordinator. Phase 1, *prepare*: the coordinator asks every participant to do the work and promise it can commit; each participant writes the changes and a prepare record durably, holds its locks, and votes yes or no. Phase 2, *commit*: if every vote was yes the coordinator writes its decision and tells everyone to commit; otherwise everyone aborts. The protocol is correct, XA-standardised, and supported by most relational databases and message brokers.

It is nevertheless avoided in service architectures, for reasons that are structural rather than fashionable. It is blocking: between prepare and commit every participant holds row locks and cannot release them — if the coordinator crashes after collecting yes-votes and before broadcasting, participants must wait, locked, until it returns, because they promised. Those locks are held across network round trips, so a 50 ms transaction becomes a 200 ms one and throughput collapses under contention. The coordinator is a single point of failure with durable state of its own. Every participant must speak the protocol, which rules out most HTTP APIs, SaaS providers, and anything you do not own — the card provider will never join your transaction. And availability compounds: a transaction spanning three participants at 99.9% each is available 99.7% of the time. 2PC has a real home — inside a single database cluster or between a database and a broker you both control — but not across service boundaries.

Two-phase commit with a coordinator crash
coordinator → Order:     PREPARE   → yes (locks held, prepare record written)
coordinator → Inventory: PREPARE   → yes (locks held)
coordinator → Payment:   PREPARE   → yes (locks held)
coordinator: write decision COMMIT … ✗ crash before sending
Order, Inventory, Payment: in-doubt. Locks held. Cannot commit (no decision), cannot abort (they promised).
                          Wait for coordinator recovery — seconds, minutes, or a human.

What replaces it: sagas, compensation, outbox

Give up atomicity across services and get it back at the business level. A saga is the sequence of local transactions — create order, reserve stock, charge card — each committed on its own, with a compensating action for each step that undoes its business effect if a later step fails: release stock, cancel order, refund. The system passes through visible intermediate states (order: pending) and ends either fully done or fully compensated. That is eventual atomicity with no locks held across the network; it costs explicit design of every undo and it cannot un-send an email. Saga Pattern covers the mechanics.

The saga needs each service to reliably do two things at once: commit its local state *and* tell the next step. Write the row and then publish the event, and a crash between the two loses the event; publish then write, and a crash leaves an event for a row that does not exist. The outbox pattern solves this with a local transaction: insert the domain row and an outbox row in the same COMMIT, then a relay process reads the outbox and publishes to the broker, marking rows sent. The relay may publish twice after a crash, so consumers must be idempotent — which they must anyway. Change-data-capture (reading the database WAL, see Replication and Read Scaling) is the outbox without the polling.

The outbox: state and event in one local transaction
1BEGIN;
2INSERT INTO orders (id, customer_id, status, total)
3  VALUES ('ord_91', 'cus_42', 'pending', 8900);
4INSERT INTO outbox (id, aggregate, event_type, payload, published_at)
5  VALUES (gen_random_uuid(), 'ord_91', 'OrderCreated',
6          '{"orderId":"ord_91","items":[{"sku":"A1","qty":2}]}', NULL);
7COMMIT;
8 
9-- relay, every 100 ms (or CDC on the WAL):
10-- SELECT * FROM outbox WHERE published_at IS NULL ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED;
11-- publish each; UPDATE outbox SET published_at = now() WHERE id = ANY($ids);
2PC vs saga vs outbox
Two-phase commitSaga + compensationOutbox (per service)
AtomicityReal, across participantsEventual, at business levelLocal: state + event together
IsolationYes (locks until commit)None — intermediate states visibleLocal only
Locks across networkYes — throughput and blocking riskNoNo
ParticipantsMust speak XA; not SaaS/HTTPAnything with an undoYour own DB + broker
Failure handlingCoordinator recovery; in-doubt stateCompensations, timeouts, pending statesRelay retries; consumers idempotent
Where it belongsInside one DB cluster / DB + broker you controlAcross servicesEvery service that emits events

Before any of this: do you need the boundary?

The cheapest distributed transaction is the one you do not have. If Order, Inventory and Payment are modules in one Modular Monolith with one database, "place order" is a local transaction again and the entire problem disappears. Splitting a flow that needs atomicity across services should be a decision made *because* of a measured organisational or scaling need, with the cost of sagas and outboxes counted in. Sharing one database between services to keep the transaction is the Microservices distributed-monolith anti-pattern: all of the network, none of the independence. See Distributed Consistency: CAP, Quorums, Consensus for what "consistent" can even mean once the data is partitioned.

Key points

  • One database gives atomicity and isolation for free; a service boundary removes both, and partial failure becomes the normal case.
  • 2PC restores atomicity with a coordinator but blocks participants with locks held across the network, has an in-doubt state on coordinator crash, and cannot include systems you do not own.
  • Sagas replace atomic rollback with committed local steps plus compensating actions; the price is visible intermediate states and designing every undo.
  • The outbox pattern makes "write state and publish event" atomic with one local transaction and a relay; consumers stay idempotent because the relay may publish twice.
  • The cheapest fix is not needing the boundary: keep flows that must be atomic inside one module and one database until a measured need forces the split.

Why one transaction cannot span three services

Why one transaction cannot span three services
Order, Payment and Inventory each own a database. Try to make one write atomic across all three.
BEGIN??Order servicePayment serviceInventory serviceOrder DBPayment DBInventory DB
Approach
locks held across network
—
blocking on coordinator
no
atomic across services
no
step
1 / 4
BEGIN on the Order service’s connection to Order DB.

Once data is split across services you have exactly two honest options: a distributed commit protocol with its locks and blocking, or a sequence of local transactions stitched together by messages (a saga), which gives up atomicity for eventual consistency.

1/4

How data moves through it

One request or event, hop by hop.

  1. 1Client → Order service: POST /orders; local transaction writes orders(status=pending) + outbox(OrderCreated); 202 to the client.
  2. 2Outbox relay → Broker: publishes OrderCreated; marks the outbox row sent.
  3. 3Broker → Inventory service: reserves stock in a local transaction with its own outbox row StockReserved (or StockUnavailable).
  4. 4Broker → Payment service: charges with an idempotency key; publishes PaymentCaptured / PaymentFailed.
  5. 5Broker → Order service: on PaymentFailed, publishes ReleaseStock and sets orders.status = cancelled; on PaymentCaptured, status = confirmed.

When to use — and when not

Use it when
  • 2PC: between resources you control inside one trust domain — a database and a broker, two databases in one cluster — where a short blocking window is acceptable.
  • Sagas: any business flow spanning services with separate databases, especially with external participants (payment providers, carriers).
  • Outbox: every service that must publish an event as a consequence of a state change; it is the default, not an option.
Avoid it when
  • 2PC across service boundaries or with SaaS participants — the locks, coordinator dependency and availability product make it worse than the problem.
  • Sagas for a flow that could simply be a local transaction in one service; the split is the cost, not the saga.
  • Any of it when the operations are naturally independent (log an analytics event, send a marketing email) — fire the event, retry on failure, no atomicity needed.

Tradeoffs

Complexity
low → high
Ops cost
low → high
Latency
low → high
Consistency
weak → strong
Scalability
poor → strong

Sagas and outboxes scale because nothing holds locks across the network; you pay with visible intermediate states and explicit compensation logic for every step.

How it fails

  • Write-then-publish without an outbox: a crash between the two loses the event and the downstream step never happens — the stock is never reserved for a paid order.
  • 2PC coordinator crash leaves participants in-doubt with locks held; other transactions on those rows block until a human resolves it.
  • A saga step succeeds but its compensation is missing or fails: the order ships but the payment was declined (the shipped-but-not-paid challenge).
  • Consumers assume the outbox relay publishes exactly once; a relay restart duplicates an event and stock is reserved twice.
  • Retrying a step that is not idempotent (charge card) during saga recovery charges twice.

How it scales

  • Sagas scale linearly with services because no lock spans the network; the coordinator (if orchestrated) is a stateless consumer of its own state table.
  • Outbox relays scale per service; CDC on the WAL removes polling overhead at high write rates.
  • 2PC throughput falls with participant count and network latency because locks are held for the full round trip — the reason it stays inside one cluster.
  • The pending-state window grows with step latency; bound each step with a timeout so a slow participant cannot hold a saga open indefinitely.

How it interacts with databases, queues, caches, APIs and external systems

  • Database (per service): local ACID transactions plus an outbox table; unique constraints on message ids for consumer deduplication.
  • Queue/log: carries the saga's events and commands; at-least-once delivery, so every consumer is idempotent.
  • External APIs: payment providers and carriers are participants that can never join a 2PC; they are steps with compensations (refund, cancel shipment).
  • Cache: never a saga participant — a cache is rebuilt from the database, not compensated.
  • API: the client sees pending and polls or subscribes; the API must never promise a final state it has not committed.
Saga Pattern →