The question this answers
Two services must both change state, or neither. Neither can see the other’s uncommitted work. What are my actual options?
By default: none across the boundary. Each service’s local transaction is atomic and durable within its own store. The *composition* of those transactions guarantees nothing — no atomicity, no isolation, no defined intermediate state. Any cross-service atomicity is something you build and pay for, and every construction weakens one of the four ACID letters.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The order service knows its own row committed. It knows it sent a request to payments and, at best, what came back. It does not know whether payments committed, whether payments will commit in a second, or whether payments committed and then got rolled back by an operator. Its belief "the order is paid" is an inference from a message, and messages are exactly what A Timeout Tells You Nothing About Whether It Happened says you cannot conclude from.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What you actually lose when the boundary appears
Inside one database, BEGIN … COMMIT gives four things at once, and because they arrive together it is easy to forget they are four. Atomicity: all writes land or none do. Consistency: declared constraints hold at commit. Isolation: nobody observes the half-written state. Durability: once committed, it survives a crash. Database owns how each of those is implemented.
Split the same work across two services with two stores and you keep durability, you keep per-service atomicity, and you lose the other two *across the boundary*. There is no lock manager that spans both, no undo log that covers both, and no single point that can be asked "is this transaction committed?" — because there is no such transaction. There are two transactions and some hope.
This is not a tooling gap. It follows from No Shared Memory: Every Node Sees a Copy: the two commit decisions happen on two machines and are separated by a network that can lose, delay, reorder and duplicate the messages between them. A protocol can make the two decisions *agree*; nothing can make them *simultaneous*.
Why you cannot just hold a transaction open
The obvious move is to keep each local transaction open until all three succeed, then commit them together. This is exactly what Two-Phase Commit: Buying Atomicity With a Promise does, and it is worth understanding *why* it is not the default rather than assuming it is unavailable.
Holding a transaction open holds locks. Locks held across a network are held for the duration of the *slowest* participant plus network time, not for the duration of a local write — microseconds become hundreds of milliseconds, and the lock contention that was invisible at 200µs becomes the system’s dominant bottleneck. Worse, the holder can now be a machine that has crashed, and the lock outlives it.
And it presumes something usually false: that every participant offers a prepare/commit interface and is willing to surrender its unilateral right to abort. A third-party payment processor does not. A REST API does not. An SMTP server does not. The set of resources that can join a two-phase commit is much smaller than the set of things a business transaction touches.
- Duration — cross-service locks are held for network-scale time, not memory-scale time.
- Ownership — a participant must expose prepare/commit and honour a promise. Most services do not.
- Trust — you are asking another team’s service to hold resources hostage to your workflow.
- Availability — the composite is now less available than any part: it needs *all* of them up at once.
The four honest options, cheapest first
The most valuable move is the first one, and it is the one most often skipped: question the boundary. An invariant that must be atomic is evidence that the data on both sides belongs to one owner. If "order created" and "payment recorded" must never disagree, they may want to be one service with one store — see Where the Boundary Goes and Exactly One Component Owns Each Piece of State. Merging two services is cheaper than operating a saga forever.
The second is to question the invariant. "The order must be paid before it exists" is often a restatement of a UI constraint, not a business rule. Businesses run on eventual consistency constantly: hotel bookings overbook and compensate, banks post transactions asynchronously and reconcile overnight. If the business already tolerates a window, do not spend a protocol closing it. Start From the Invariant, Not From the Architecture is about deciding which invariants genuinely need coordination.
Only then do the protocols matter: 2PC if all participants sit inside one administrative domain with a reliable coordinator and short transactions, and a saga if they do not. Those are the last two lessons’ subjects, and the rest of this module.
| Option | Atomicity | Isolation | Availability | Where it fits |
|---|---|---|---|---|
| Collapse the boundaryprotocol | Full — one txn | Full | One store’s availability | The invariant is really one owner’s |
| Relax the invariantassumption | Not required | Not required | Highest | Business already tolerates a window |
| Two-phase commitprotocol | Yes, if coordinator survives | Yes, with 2PL held through prepare | Lowest — needs all up | One trust domain, short txns |
| Saga + compensationprotocol | Eventual, semantic | None — intermediate states visible | High — per-step | Across trust domains, long-running |
The dual-write problem hiding underneath
Even a single service usually has two stores: its database and its message broker. "Commit the row, then publish the event" is a cross-service transaction wearing a disguise, and it fails the same way — commit succeeds, publish fails, and the rest of the system never learns. Reverse the order and you publish an event for a row that was never committed.
The standard fix is the transactional outbox: write the event into a table *in the same local transaction* as the state change, and let a separate relay read that table and publish. That converts a distributed problem into a local one plus at-least-once delivery, which is a problem you already know how to solve with idempotent consumers. Choreography: The Workflow Nobody Wrote Down depends on it entirely.
1-- Broken: two systems, no atomicity between them.2BEGIN;3 UPDATE orders SET status = 'confirmed' WHERE id = $1;4COMMIT;5publish("order.confirmed", $1); -- may never run; may run twice6 7-- Outbox: one local transaction, one system.8BEGIN;9 UPDATE orders SET status = 'confirmed' WHERE id = $1;10 INSERT INTO outbox (id, topic, payload)11 VALUES (gen_random_uuid(), 'order.confirmed', $2);12COMMIT;13-- A relay polls or tails the WAL and publishes at-least-once.14-- Consumers must be idempotent. They always had to be.Key points
- ACID is scoped to one transaction manager over resources it controls; the boundary, not the database, is what ends it.
- Across services you keep durability and per-service atomicity, and lose atomicity and isolation for the composition.
- Holding local transactions open across a network converts microsecond locks into network-latency locks held by machines that may be dead.
- Question the boundary and the invariant before choosing a protocol — merging two services beats operating a saga forever.
- "Write to the DB then publish an event" is a cross-service transaction in disguise; the outbox turns it back into a local one.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A business operation is decomposed into steps, each owned by a different service with its own store.
- • Each service executes its step inside a local ACID transaction and commits independently.
- • Between commits, the system is in a state no single store represents: order created, payment not yet taken.
- • Some mechanism — a coordinator, an event chain, or a human — must drive the remaining steps or undo the completed ones.
- • Until that mechanism finishes, readers of either store see a partial truth, and no error is raised anywhere.
- • A step commits and its confirmation is lost, so the caller believes it did not happen (A Timeout Tells You Nothing About Whether It Happened).
- • A step is retried and executes twice because it was not idempotent.
- • The process driving the steps crashes between two of them, leaving no one responsible for finishing.
- • A downstream service reads the intermediate state and acts on it — ships the goods for an unpaid order.
- • The services disagree permanently and nothing detects it, because neither is in an error state.
- • Orphaned charges: the operator sees a payments table with successful charges whose
order_iddoes not exist in the orders service. No alert fired; both services report 100% success rates. - • Stuck intermediate state: orders sit in
awaiting_paymentfor days. The dashboard shows healthy p99 and zero errors, because the failing thing is a step that was never attempted. - • Phantom events: consumers act on an
order.confirmedevent for an order the orders DB rolled back — the classic publish-before-commit dual write. Operator sees shipping labels for orders that do not exist. - • Silent divergence discovered by finance: month-end reconciliation finds the ledger and the order book differ by a few hundred rows, and nobody can say when the drift started because there is no signal for it.
- • Zero coordination is the default and it buys maximum availability and zero atomicity.
- • Any atomicity across the boundary requires at least one round trip of agreement before any participant can commit — that is Coordination Couples Availability paid as availability, not just latency.
- • The composite availability of an all-must-agree protocol is the *product* of participant availabilities: three services at 99.9% give 99.7% for the transaction, before the coordinator is counted.
- • A saga requires no synchronous agreement, but requires durable state somewhere that outlives every participant — which is a coordination point of a different shape.
- • Each committed local step remains durable regardless of what happens to the others. Nothing un-commits itself.
- • Cross-service invariants are unenforced for the whole window, and the window has no upper bound unless something enforces one.
- • Reads are unrestricted: any service may observe the half-finished state and act on it, because no isolation mechanism spans the boundary.
- • Detect: a reconciliation job that joins the two sides on a shared correlation id and reports rows present on one side only, with an age.
- • Contain: make forward progress possible by giving every step a durable record of intent before it runs, so a crashed driver can be resumed rather than restarted.
- • Recover: re-drive incomplete operations from that durable record, with idempotent steps so re-driving is safe (Idempotent Is a Property of the Whole Effect, Not the Write).
- • Reconcile: for operations that cannot complete, compensate the committed steps — and note that compensation is a new business action, not an undo (A Refund Is Not a Rollback).
- • Verify: alert on the *age of the oldest unreconciled item*, not on error rate. This failure class produces no errors.
- • Count and age of business objects in non-terminal states, bucketed by state. A state machine with no aging metric hides every stuck workflow.
- • Reconciliation delta between each pair of services that share an invariant, published as a gauge, not a log line.
- • Rate of steps whose outcome was ambiguous (timeout rather than a definite response) — the population from which divergence is drawn.
- • Outbox lag: rows in the outbox table older than N seconds. A growing outbox is an event stream that has silently stopped.
- • Recognising the problem early is what helps — the cost of a cross-service invariant is paid at design time or paid forever in operations.
- • When services genuinely have different owners, lifecycles, scaling profiles or trust levels, the boundary earns its cost and the transaction problem is the price.
- • When one side is a third party you cannot enrol in any protocol, this framing tells you immediately that saga-with-compensation is your only option, and you can design for it rather than discover it.
- • When the boundary exists only because of an organisational chart or a fashion for microservices. Then you have bought a distributed transaction problem in exchange for nothing (The Distributed Monolith: All of the Cost, None of the Autonomy).
- • When the invariant is enforced synchronously across three services on the request path, so the checkout page cannot render unless all three are healthy.
- • When the team responds to the complexity by adding a shared database so both services can transact — which removes the transaction problem and creates a coupling problem that is harder to undo (The Shared Database: An Honest Trade, Not a Prohibition).
- • Merge the services so the invariant is local. Usually the correct answer and rarely the considered one.
- • Relax the invariant to eventual with an explicit reconciliation process and a stated convergence window.
- • Make one service the Source of Truth: The Question Every Inconsistency Incident Is Really Asking and let the other hold a derived, possibly stale copy — the invariant then holds by construction at exactly one place.
- • Keep the operation synchronous but make it a single call to one service that owns all three writes, with the other services reduced to read models.
- • Use a durable workflow engine so the driver’s crash-safety is a solved problem rather than a thing you re-implement per workflow.
Commit the row, publish the event — and crash in between
BEGIN;
UPDATE orders SET status = 'confirmed' WHERE id = $1;
COMMIT;
publish("order.confirmed", $1); -- may never run| Atomicity | Isolation | Availability | When it fits | |
|---|---|---|---|---|
| Collapse the boundaryprotocol | Full — one transaction | Full | One store's availability | The invariant is really one owner’s data |
| Relax the invariantassumption | Not required | Not required | Highest | The business already tolerates a window |
| Two-phase commitprotocol | Yes, if the coordinator survives | Yes — locks held through prepare | Lowest: needs every participant up | One trust domain, short transactions |
| Saga + compensationprotocol | Eventual, semantic | None — intermediate states are visible | High, per step | Across trust domains, long-running |
What people believe, and what is true
A distributed transaction is just a normal transaction with more participants.
A normal transaction has one decision point. A distributed one has several, separated by a network, which is why an entire family of protocols exists and why all of them trade something away.
If both services use the same database engine, we can just use a transaction.
Only if they use the same database *instance* and the same connection — at which point they are not separate services in any meaningful sense, and you have a The Shared Database: An Honest Trade, Not a Prohibition.
We commit locally then publish an event, so the event is guaranteed.
The commit and the publish are two systems with no atomicity between them. The event can be lost after a successful commit, or published for a transaction that rolled back. That is the dual-write problem, and the outbox exists because of it.
Eventual consistency means the data will be correct eventually.
It means replicas converge if writes stop. It says nothing about a saga that halted halfway — that state is not eventually consistent, it is permanently wrong until something acts on it.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One database gives all-or-nothing free. Two services do not, because two machines make two commit decisions. Before choosing a protocol, ask whether the two pieces of data should be owned by one service.
Practical
Give every cross-service operation a correlation id, a durable record of intent written before the first step, and a reconciliation job that joins both sides on that id. Alert on the age of the oldest unreconciled row. Replace every "commit then publish" with an outbox.
Advanced
The choice is between blocking and exposure. 2PC keeps isolation by holding locks through an uncertainty window, so it blocks on coordinator failure. Sagas keep availability by committing each step immediately, so intermediate states are visible and must be handled at the business level. There is no protocol that keeps both, because keeping both requires the two commit decisions to be one decision.
Apply it
- 🔧 Take an existing two-service workflow in your system and write down its invariant precisely. Then find the window in which it is violated and measure how long that window actually is in production.
- 🔧 Implement an outbox relay and deliberately kill it mid-publish. Verify that consumers see the event at least once and that a duplicate is harmless.
- ⚡ A booking system creates a reservation, charges the customer and emails a confirmation. The email service is a third party that cannot participate in any commit protocol. Design the ordering of the three steps and justify it.
- ⚡ Finance reports that the ledger and the order book disagree by 0.02% of rows. Neither service has logged an error in six months. Where do you look first?
- 💬 Checkout must create an order, charge a card and reserve stock. Three services, three databases. Walk me through what happens when the stock reservation fails after the card is charged.
- 💬 A team proposes "we write to Postgres and then publish to Kafka". What is wrong with that, and what would you propose instead?
- 💬 When would you argue for merging two services rather than adding a saga between them?
- 💬 Your three services are each 99.9% available. What is the availability of a synchronous operation requiring all three, and what would you do about it?