The question this answers
If I cannot hold a transaction open across services, what does the alternative actually guarantee — and what does it stop guaranteeing?
Each step is atomic and durable in its own store. The saga as a whole guarantees only eventual semantic completion: it reaches either "all steps committed" or "all committed steps compensated" — and only if the saga’s state is durable, every step and compensation is idempotent, and compensations are retried until they succeed. There is no isolation: intermediate states are readable by everyone, including other sagas. There is no atomicity at any instant, only convergence.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
Each participating service knows only that its own local transaction committed. It does not know which step of the saga it is, whether earlier steps are still valid, or whether a later step will fail and cause its work to be compensated. Whatever knowledge of "the saga" exists lives in the orchestrator’s state store or is scattered across an event log — never in the participant. A participant asked "is this reservation going to survive?" cannot answer.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The shape: commit as you go, undo backwards
A saga is a sequence T₁ … Tₙ of local transactions, each with a compensating transaction C₁ … Cₙ₋₁. Run forward; if Tₖ fails, run Cₖ₋₁ … C₁ backwards. Each Tᵢ commits the instant it completes, so no locks are held between steps and no participant waits on any other.
That is the entire idea, and its consequences are almost all about what "commits immediately" means. Once T₁ commits, the world can see it. A customer refreshes the page and sees the order. A reporting job counts it. Another saga reads the reserved inventory and makes its own decisions. None of them know the saga may be about to unwind.
The name and the design come from Garcia-Molina and Salem’s 1987 paper, which was about *long-lived* transactions inside one database — transactions whose duration made holding locks unacceptable. The distributed-systems reading came later, but the motivation is the same in both: the lock hold time is the problem, and giving up isolation is the price of eliminating it.
The missing I: what "no isolation" costs concretely
ACID without the I is sometimes written ACD, and it is a fair description. Everything a database’s isolation level protects you from is back on the table, across services, with no engine to help. The classic anomalies return in distributed clothing.
Dirty reads: another saga (or a user, or a report) reads state that will later be compensated. The report is wrong, the user is confused, and the second saga may have made an irreversible decision based on it. Lost updates: two sagas both read a value, both write it, and one overwrites the other — the compensation makes this worse, because a naive compensation writes back a remembered old value and destroys an unrelated concurrent change. Non-repeatable reads within one saga: step 4 re-reads what step 1 saw and gets something different.
Garcia-Molina’s paper anticipated this and proposed *countermeasures* — application-level substitutes for the isolation the engine no longer provides. They are the practical toolkit, and it is worth knowing them by name because they are what you will actually write.
- Semantic lock — mark the record with an in-progress state (
PENDING_PAYMENT) so other readers know not to trust it, and other writers know to back off. A lock in the business model rather than the lock manager. - Commutative updates — write
balance = balance - 50, neverbalance = 950. Then order does not matter and a compensation cannot clobber a concurrent change. - Pessimistic view — reorder steps so the risky, hard-to-compensate step happens where the exposure window is smallest.
- Re-read value — before updating, re-read and verify nothing changed since you last looked; abort the saga if it did. Optimistic concurrency, applied across services.
- Version file — record the operations applied to a record so that out-of-order arrivals can be reordered or detected.
- By value — route low-risk requests through the saga and high-risk ones through a real distributed transaction. The dynamic-choice countermeasure, and the honest admission that sagas are not always the answer.
| Property | Single transaction | Saga | Who provides it now |
|---|---|---|---|
| Atomicityprotocol | Instant, all-or-nothing | Eventual, semantic | Saga driver + compensations |
| Consistencyassumption | Constraints at commit | Per-step only | Application invariants + reconciliation |
| Isolationprotocol | Engine-provided | None | You — countermeasures in business logic |
| Durabilityprotocol | Per commit | Per step, unchanged | Each participant’s own store |
The saga log is the whole design
A saga without durable state is not a saga; it is a sequence of calls that happens to work when nothing fails. The essential component is a durable record of what has been attempted and what has completed, written *before* each step is dispatched, so that a driver which crashes mid-saga can be resumed by another process rather than leaving the work orphaned.
The ordering rule is the same one 2PC uses for its decision log, and for the same reason. Persist "about to run step 3" *then* run step 3. A crash between the two means step 3 may or may not have run, which is A Timeout Tells You Nothing About Whether It Happened — and the resolution is that step 3 must be idempotent, so resuming can simply re-run it. Persist after running instead, and a crash loses all knowledge that step 3 happened, which no amount of idempotence recovers.
This is why "at-least-once execution plus idempotent steps" is not a nice-to-have in saga design. It is the mechanism by which crash recovery works. A saga whose steps are not idempotent has no safe resume, and its only recovery is a human reading logs.
1async function runStep(sagaId: string, step: Step) {2 // 1. Durable intent BEFORE the effect. A crash after this leaves an3 // ambiguous step, which resume handles by re-running it.4 await sagaLog.record(sagaId, step.name, 'STARTED', step.idempotencyKey)5 6 // 2. The effect. Carries the key so a re-run collapses into the original.7 const result = await step.invoke(step.idempotencyKey)8 9 // 3. Durable outcome. A crash before this means resume re-runs step 2,10 // which is safe precisely because of the key.11 await sagaLog.record(sagaId, step.name, 'COMPLETED', step.idempotencyKey, result)12 return result13}14 15// Resume after any crash: replay the log, re-run anything not COMPLETED.16// This is only correct because every step is idempotent under its key.When a saga is the right shape
Sagas fit when the steps genuinely belong to different owners, when the operation is long-lived, and when the business already has a notion of undoing things — because it almost always does. Refunds, cancellations, reversals, credit notes and restocking are not engineering inventions; they exist because businesses have always had to unwind partially completed work. A saga is that process, written down.
They fit badly when the invariant is hard — when there is no acceptable state in which it is violated even briefly. "Two people must never book the same seat" is not a saga; it is a uniqueness constraint that needs Distributed Uniqueness: One Name, Many Shards or a single owner. A saga will let both bookings through and then compensate one, which is a business decision (airlines make it deliberately) and should be made consciously rather than discovered in production.
And they fit badly when the number of steps grows. Compensation logic is O(n) code paths that run rarely and are therefore under-tested. A five-step saga has five compensations, each of which can fail, each of which must be idempotent, and each of which must work against state that other sagas have since modified. That is the real cost, and it is paid in maintenance forever, not in latency.
Key points
- A saga is a sequence of local transactions, each committed immediately, each with a compensating action.
- It never blocks and holds no cross-service locks — that is the entire benefit.
- It surrenders isolation completely: intermediate state is visible and other sagas can act on it.
- The lost isolation must be replaced by application-level countermeasures: semantic locks, commutative updates, re-read checks.
- The durable saga log, written before each step, is what makes crash recovery possible; steps must be idempotent for resume to be safe.
- Cost is paid in rarely-exercised compensation code, not in latency.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Decompose the operation into steps that each fit inside one service’s local transaction.
- • Define a compensating action for each step that can be reversed, and identify the ones that cannot.
- • Record the saga’s intent durably before dispatching a step; record its outcome durably after.
- • Run steps forward. Each commits locally and becomes immediately visible.
- • On a step failure, run the compensations for the completed steps, normally in reverse order.
- • Retry both steps and compensations until they succeed or the saga is escalated to a human; never abandon a saga silently.
- • Mark the saga terminal — COMPLETED or COMPENSATED — and stop.
- • A step times out with an unknown outcome, so the driver does not know whether to advance or compensate.
- • The driver crashes between two steps, leaving committed work with nobody responsible for it.
- • A compensation fails, leaving the saga unable to go forward or backward.
- • A compensation arrives before the step it compensates, because of a retry and reordering.
- • Another saga reads the intermediate state and commits an irreversible decision on it.
- • A step succeeds after the driver gave up on it, applying an effect to an already-compensated saga.
- • Customer charged for a cancelled order: the operator sees a payments record with no matching active order, and a support ticket, before any monitoring notices. The saga did compensate — or would have, if the refund step had not been failing for six hours.
- • Sagas stuck in COMPENSATING: a dashboard of saga states shows a growing bucket in COMPENSATING with ages in hours. Every service involved reports healthy. The stuck compensations are usually all hitting the same one downstream dependency.
- • Orphaned sagas after a deploy: the driver was restarted and its in-memory state lost because the saga log was written after the step rather than before. Operator sees inventory reserved against orders that no longer exist and stock levels drifting from the physical count.
- • Anomaly cascade: a reporting job counted revenue from sagas that later compensated, so the finance dashboard and the ledger disagree — and the discrepancy varies with the failure rate of step 3, which nobody connects to the finance number.
- • Compensation clobber: the compensation wrote back a remembered old value and destroyed a concurrent legitimate update. The operator sees a customer’s address revert to a previous value with no audit entry explaining it.
- • No synchronous coordination between participants at all — no participant ever waits for another. That is the availability win.
- • Coordination is replaced by *durable state plus retries*: someone must remember the saga and keep driving it, which is a different cost with a different failure surface.
- • Ordering coordination remains where steps are dependent: step 3 cannot start before step 2 has definitely committed, so the saga’s latency is the sum of its steps, not the max.
- • The countermeasures reintroduce coordination selectively — a semantic lock is a coordination point you chose deliberately, applied to one record rather than to the whole transaction.
- • Every committed step stays committed until explicitly compensated. Nothing rolls back on its own.
- • The system is in a state that violates the cross-service invariant, and remains there for as long as it takes to compensate — which is unbounded if the compensation keeps failing.
- • Reads are unrestricted throughout: there is nothing to prevent any observer from seeing and acting on the inconsistent state.
- • Per-service correctness is untouched; each store is internally consistent and would pass its own constraint checks.
- • Detect: track saga state and age. The signal is not error rate — it is the count of sagas in non-terminal states older than the expected completion time.
- • Contain: cap concurrent sagas and make the driver refuse to start new ones when the compensating backlog grows, so a broken downstream does not accumulate an unbounded repair queue.
- • Recover: resume from the saga log. Re-run any step whose outcome is unknown, relying on idempotence to make the re-run harmless.
- • Reconcile: for sagas that cannot complete or compensate automatically, route them to an operator queue with the full step history — and make that queue a first-class product surface, not a log grep.
- • Verify: reconcile each pair of participating services on the saga correlation id, and alert on any row present on one side only.
- • Saga state distribution and age histogram per state. A bucket that grows and does not drain is the whole story.
- • Compensation invocation rate and compensation *failure* rate, tracked separately from forward-step failures — they have different urgencies.
- • Time from saga start to terminal state, p50 and p99. The p99 is the length of your isolation-anomaly window.
- • Count of sagas escalated to human intervention, which is the true measure of whether the design is working.
- • Per-step ambiguous-outcome rate (timeouts), because those are the inputs to every duplicate and every orphan.
- • Long-running operations where holding any lock is unacceptable — anything spanning human think time, external approval, or a shipment.
- • Steps owned by different teams or external parties who will never join a commit protocol.
- • Where the business already has reversal semantics — refunds, cancellations, restocking — so compensations are natural business actions rather than inventions.
- • Where availability of each step matters more than the invariant being instantaneously true, which is most of e-commerce.
- • Hard invariants that must never be violated even briefly — uniqueness, non-negative balances with legal consequences, safety interlocks.
- • Sagas with many steps, where compensation code becomes a large, rarely-executed, under-tested surface.
- • Where a step is genuinely irreversible and sits early in the sequence, so failure downstream leaves nothing to do (A Refund Is Not a Rollback).
- • Where intermediate state is exposed to users or to other automated systems that will act on it irrevocably.
- • As a replacement for a millisecond-scale intra-cluster transaction, where 2PC would have been simpler and safer.
- • Collapse the steps into one service with one transaction, when the data really has one owner (Atomicity Stops at the Process Boundary).
- • Two-phase commit, when all participants are inside one trust domain and the transaction is short (Two-Phase Commit: Buying Atomicity With a Promise).
- • Reorder so the operation becomes a single atomic step plus asynchronous follow-ups that are retriable but never compensated.
- • Reservation pattern: take a short-lived, expiring hold instead of a commit, so "compensation" is simply letting the hold lapse (Leases: Authority With an Expiry Date).
- • Accept the inconsistency and reconcile in batch, where the volume is low and the business impact of a rare mismatch is smaller than the cost of the machinery (Reconciliation Is a Component, Not a Cleanup Script).
Order → Payment → Inventory → Shipping, and the compensations that follow
What people believe, and what is true
A saga gives you eventual atomicity, which is basically atomicity with a delay.
Atomicity means no observer ever sees a partial state. A saga guarantees the opposite: observers *will* see partial states. What it gives you is eventual convergence to an acceptable state, which is a much weaker and differently-shaped property.
Sagas are strictly better than 2PC because they do not block.
They exchange blocking for permanently absent isolation and a large body of compensation code. For a short intra-cluster transaction that is a bad trade.
If a step fails, the saga rolls back.
It compensates. Committed work is not erased; new work is done to counteract it, and that new work is visible (A Refund Is Not a Rollback).
The saga library handles correctness for me.
A library handles orchestration, persistence and retries. It cannot write your compensations, cannot know which reads are unsafe, and cannot decide which of your invariants tolerate a window.
We do not need the log; the orchestrator keeps the state in memory.
Then a restart orphans every in-flight saga. The durable log written before dispatch is the mechanism, not an optimisation.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Do the work in steps, commit each one immediately, and if a later step fails, run compensating actions for the earlier ones. Nothing blocks; everyone can see the half-done state.
Practical
Write the saga state durably before each step, give every step an idempotency key, and make resume re-run anything not marked complete. Expose saga state and age as a metric. Put an operator queue behind the sagas that cannot self-resolve, and treat its depth as a health signal.
Advanced
The interesting design work is replacing isolation. Enumerate the anomalies your data actually permits — dirty read, lost update, non-repeatable read — and choose countermeasures per anomaly: semantic locks where readers must be warned, commutative updates where concurrent writers exist, re-read-and-verify where a stale decision is expensive. Then decide which steps are compensatable, which is the pivot, and which are merely retriable; that classification determines the whole ordering.
Apply it
- 🔧 Take a three-step saga and deliberately kill the driver between each pair of steps. Verify that resume produces the same final state in every case.
- 🔧 Run two sagas concurrently against a shared record with a naive compensation that restores a remembered old value, and demonstrate the lost update. Then fix it with a commutative update.
- ⚡ Your saga has been at 99.98% completion for months. Finance reports a growing discrepancy. Where do the missing 0.02% live and how do you find them?
- ⚡ Product asks why a customer briefly saw an order confirmation for an order that then vanished. Explain the mechanism and propose what you would change.
- 💬 What does a saga guarantee, precisely? Name what it does not guarantee.
- 💬 A saga charges a card in step 2 and reserves stock in step 3. Stock fails. Describe everything an outside observer could have seen during the window.
- 💬 Two sagas run concurrently against the same inventory record. What can go wrong, and which countermeasure would you apply?
- 💬 Where does the saga’s state live, and what exactly breaks if it is not durable?