The question this answers
My saga failed at step 3, so I will undo steps 1 and 2. Why is "undo" the wrong word, and what does it cost me to use it?
A compensation restores the business invariant, not the prior state. The system converges to a state the business considers acceptable — net-zero money moved, stock available again, order cancelled — while the record of what happened remains, and remains visible. There is no guarantee that any observer who saw the intermediate state can be made to un-see it, and no guarantee that a compensation exists at all for a given effect.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The service running a compensation knows only that it was asked to counteract an effect it may or may not have applied. It does not know whether the original step committed, whether some other saga has since modified the same record, whether the customer already saw the intermediate state, or whether an earlier attempt at this same compensation already succeeded. Every compensating action is therefore a remote call under A Timeout Tells You Nothing About Whether It Happened, made against state that has moved on.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What a rollback actually does, and why you cannot have it
A database rollback works because the engine controls the entire timeline. Nothing was visible outside the transaction (isolation), the undo information is in the same log as the redo information (atomicity), and no external party ever observed the intermediate state. The engine can therefore restore the pre-transaction state and honestly claim nothing happened, because from every legal observer’s point of view, nothing did.
None of those three conditions holds across services. The intermediate state *was* visible — that is the defining property of a saga. The undo information lives in a different system from the redo. And an external party — the customer, the card network, a downstream consumer — may already have observed and acted on it.
So the compensating action cannot be an undo. It is a new local transaction, executed later, that produces a *new* effect intended to counteract the first. charge(100) compensated by refund(100) leaves a balance of zero and a ledger with two rows. That ledger is not a bug to be hidden; in most regulated domains it is legally required. The system state converges; the history does not.
| Rollback | Compensation | |
|---|---|---|
| Visibility of the originalprotocol | Never visible outside the transaction | Visible, possibly for a long time |
| Trace left behindprotocol | None — as if it never happened | Two entries — it happened and was counteracted |
| Can it fail?protocol | No — the engine guarantees it | Yes — it is a distributed call like any other |
| Needs idempotence?protocol | No | Yes — it will be retried |
| Orderingassumption | Instantaneous | May arrive out of order, or before the original |
| Always available?protocol | Yes, before commit | No — some effects have no compensation |
The compensation is itself a distributed operation
This is the consequence teams discover late. A compensating action is a remote call. It can time out with an unknown outcome. It can be retried and applied twice. It can be delivered out of order. It can fail permanently because the compensating service is down, or because the operation is no longer legal — the refund window closed, the account was frozen, the coupon expired.
So every compensation needs the full apparatus that every other step needs: its own idempotency key, its own retry policy with backoff, its own dead-letter path, its own monitoring. A team that carefully makes forward steps idempotent and then writes await paymentService.refund(orderId) in the catch block has built a saga whose recovery path is less reliable than its happy path — which is precisely backwards, because the recovery path runs exactly when things are already going wrong.
And a failed compensation is worse than a failed step, because the saga can now go neither forward nor backward. A failed forward step has an obvious response: compensate. A failed compensation has no protocol-level response at all. It is the saga’s equivalent of 2PC’s blocking window, except that no coordinator recovery will resolve it — only a human, or a business decision to accept the loss.
1// Wrong: the recovery path is less robust than the happy path.2try {3 await reserveStock(order)4} catch (e) {5 await paymentService.refund(order.id) // no key, no retry, no DLQ6 throw e7}8 9// Right: the compensation is a first-class durable step.10await sagaLog.record(sagaId, 'compensate:charge', 'STARTED', compKey)11await retry(12 () => paymentService.refund({13 // Derived deterministically, so every retry — and every resume after a14 // crash — presents the SAME identity to the payment service.15 idempotencyKey: `comp:${sagaId}:charge`,16 // Relative, not absolute: never "set balance to what it was".17 amount: chargedAmount,18 reason: 'saga_compensation',19 }),20 { attempts: Infinity, backoff: 'exponential', jitter: true,21 onExhausted: () => escalate(sagaId, 'COMPENSATION_STUCK') },22)23await sagaLog.record(sagaId, 'compensate:charge', 'COMPLETED', compKey)Compensating against state that has moved on
The naive compensation restores a remembered value: "before my step, the address was X, so set it back to X." Between the step and the compensation, another saga legitimately changed the address to Y. Your compensation now silently destroys Y. This is a lost update, and it is the single most common compensation bug.
The fix is to make effects and compensations relative and commutative wherever possible. balance -= 50 compensated by balance += 50 is correct no matter what else happened in between, and no matter what order the operations land in. stock_reserved += 1 compensated by stock_reserved -= 1 likewise. Absolute assignment is only safe when guarded by a version check that fails loudly rather than overwriting (Two Writes, No Order, One Answer Required and Version Vectors: Making the Conflict Visible are the same reasoning in a replication setting).
The second ordering hazard is stranger and catches people out: a compensation can arrive before the thing it compensates. The driver times out on step 2, decides to compensate, and issues the refund — while step 2’s original charge is still in flight and commits afterwards. The compensating service is asked to reverse something that has not happened yet. This is not hypothetical; it is the ordinary consequence of A Timeout Tells You Nothing About Whether It Happened plus retries.
Therefore a compensation must be tolerant of a missing original. "Cancel reservation R" where R does not exist must be a durable, recorded no-op that also poisons R so that a late-arriving create is rejected — not a 404 that the driver treats as a failure and not a silent success that lets the zombie original commit. The compensating service needs a tombstone, not an error.
Effects that cannot be compensated
Some effects have no counteraction, and no amount of engineering creates one. An email that was delivered has been read. A push notification appeared on a lock screen. A pallet left the warehouse on a truck. A tweet was seen and screenshotted. Data was disclosed to a third party. A trade was executed on an exchange. A physical door was unlocked.
For these there is no compensation, only a *follow-up*: a correction email, a recall request, a return label, an apology credit. These are new business actions with their own cost and their own failure rate, and — critically — they do not restore the invariant. They negotiate a different acceptable state. "We sent you an order confirmation in error" is not the same outcome as never having sent it, and pretending otherwise in a design document is how teams end up promising customers something the system cannot deliver.
The design rule that follows is the most actionable idea in this module: order the saga so that non-compensable effects happen last. Every step before them can be undone if a later step fails; once you cross into the irreversible region, the only legal direction is forward.
| Effect | Compensable? | What the "compensation" really is | What the customer sees |
|---|---|---|---|
| DB row writtentypical | Yes, cleanly | A counter-write | Nothing, if never read |
| Card chargedtypical | Yes, semantically | A refund — a new transaction | Two lines on the statement |
| Stock reservedtypical | Yes | Release the reservation | Nothing, unless they saw "1 left" |
| Coupon redeemedassumption | Partially | Reissue a new coupon | A different coupon code |
| Email sentprotocol | No | A correction email | Two emails, and confusion |
| Push notification deliveredprotocol | No | Nothing — it was already read | A notification about an order that does not exist |
| Physical shipment dispatchedprotocol | No | A recall and a return label | A parcel they must send back |
| Data disclosed to a third partyprotocol | No | A deletion request you cannot verify | Potentially a breach notification |
Compensatable, pivot, retriable — the ordering that follows
Classify every step in a saga into one of three kinds. Compensatable: it has a real compensation and can be undone semantically. Pivot: the point of no return — the step after which the saga must complete. Retriable: it comes after the pivot, it is guaranteed to eventually succeed given enough retries, and it is never compensated.
The pivot is often the irreversible step itself, or the last step that can still fail for business reasons. Once you order the saga as *compensatable steps → pivot → retriable steps*, two useful properties fall out. Before the pivot, any failure unwinds cleanly. After the pivot, there is nothing to unwind, so the only obligation is to keep retrying until every remaining step succeeds — which means those steps must not be able to fail for business reasons at all, only for transient ones.
This classification is a design constraint, not a description. If your "retriable" step can return "insufficient funds", it is not retriable and the saga is mis-ordered. Fixing that usually means moving a check earlier — validate before the pivot, act after — which is exactly why the classification is worth doing on paper before writing code.
- Before the pivot — every step must have a working, idempotent, retried compensation.
- The pivot — the last step that may fail for a business reason. After it, business failure is not an allowed outcome.
- After the pivot — steps must be retriable forever and must never need compensating. Put irreversible effects here, and put the most irreversible one last.
- Emails and notifications belong at the end. Almost every "we sent a confirmation for an order that was cancelled" incident is a saga with the email in the wrong position.
What you actually owe the customer
Because compensation is visible, part of the design is not technical. If a charge and a refund will both appear on a statement, someone must decide whether the customer is told, and what the wording is. If a confirmation email will be followed by a cancellation email, product must own that pair. If a reservation lapses, the UI must not have promised it was guaranteed.
This is the sense in which "compensation is not rollback" is a product statement as much as an engineering one. A rollback needs no communication because nobody saw anything. A compensation always has an audience, and designing the saga without designing what that audience sees is how a technically correct system generates support tickets.
The useful test when reviewing a saga design: for each compensation, ask "who will notice, and what do they see?" If the answer is "nobody" the step was probably cleanly compensatable. If the answer names a human, you have found the part of the design that needs a product decision, not a retry policy.
Key points
- A rollback erases; a compensation adds. The money moved, and the ledger keeps both entries.
- Rollback is possible only because the engine controls visibility, undo information and observers. None of that holds across services.
- A compensating action is an ordinary distributed call: it can time out, duplicate, reorder, and fail permanently — so it needs its own key, retry, DLQ and monitoring.
- A failed compensation blocks the saga in both directions and has no protocol-level resolution.
- Compensations must be relative and commutative, never a restore of a remembered absolute value, or they destroy concurrent updates.
- A compensation can arrive before the effect it compensates; it must write a tombstone, not return 404.
- Some effects — email, notification, shipment, disclosure — cannot be compensated at all. Order the saga so those happen last.
- Classify steps as compensatable → pivot → retriable, and let that classification drive the ordering.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The saga driver observes a failure and decides to unwind the completed steps.
- • For each completed step, in dependency order, it dispatches the compensating action with a deterministic identity derived from the saga id and step name.
- • The compensating service applies a new local transaction that counteracts the original effect — a refund row, a released reservation, a status change to CANCELLED.
- • The compensating service records that this compensation was applied, so a retry is a no-op, and so a late-arriving original can be rejected.
- • The driver records each compensation durably before and after dispatching it, exactly as it does for forward steps.
- • If a compensation cannot succeed after its retry budget, the saga is escalated rather than abandoned — there is no automatic resolution.
- • The saga terminates in COMPENSATED, and the record of everything that happened remains queryable.
- • The compensation times out with an unknown outcome, and retrying may refund twice.
- • The compensation is delivered before the original effect, which has been delayed in the network.
- • The compensation is no longer legal: the refund window closed, the account is frozen, the shipment already left.
- • The compensating service is permanently unavailable, so the saga can go neither forward nor back.
- • The compensation restores an absolute value and destroys a concurrent legitimate change.
- • The effect has no compensation at all, and the saga was designed as if it did.
- • The compensation succeeds but a downstream consumer already acted on the intermediate state irreversibly.
- • Double refund: the compensation was retried without an idempotency key. The operator sees a negative balance and two credit entries with different reference ids, and finds it during month-end, not in monitoring.
- • Sagas frozen in COMPENSATING: a dashboard bucket that grows and never drains, every entry blocked on the same downstream service. Forward traffic is healthy, so no error-rate alert fires.
- • Cancellation email arrives before the confirmation email: the compensation path was faster than the original notification queue. Support sees confused customers; engineering sees two successful sends.
- • Reverted field with no explanation: a compensation wrote back a remembered address, silently overwriting an update the customer made two minutes earlier. The audit trail shows a change with no actor.
- • Recall costs: goods shipped for a saga that later compensated. The operator sees return-label spend rising, and the root cause is a saga whose dispatch step sits before its pivot.
- • Zombie original: the compensation returned 404 for a reservation that did not exist yet, the driver marked compensation failed, and the delayed original then created the reservation. Stock is now held against a cancelled order and nothing will ever release it.
- • Compensation requires no synchronous agreement, which is why sagas remain available — but it does require durable knowledge of what to compensate, which must outlive every participant.
- • The compensating service and the original service must agree on the *identity* of the effect being counteracted, which is a shared naming decision made at design time rather than at runtime (What Counts as the Same Operation?).
- • Ordering between compensations is a dependency constraint, not a protocol requirement: independent steps may compensate concurrently, dependent ones must not.
- • The pivot is a coordination point in disguise — it is the moment the system commits to a direction, and everything after it must be designed as unconditional.
- • Committed effects remain committed until a compensation succeeds; nothing expires or reverts on its own.
- • The business invariant stays violated for the whole compensation window, and that window has no upper bound if the compensation keeps failing.
- • Non-compensable effects remain in force permanently regardless of the saga’s outcome.
- • Each participating service remains internally consistent throughout; the inconsistency exists only in the relationship between them.
- • Detect: alert on compensation failure rate and on the age of the oldest saga in a compensating state — separately from forward-step health.
- • Contain: stop starting new sagas that depend on the failing compensator, so the repair backlog stops growing while you work.
- • Recover: retry compensations with the same deterministic key indefinitely, with backoff and jitter, because a compensation that eventually succeeds is always better than one abandoned.
- • Reconcile: route exhausted compensations to a human queue carrying the full step history and the customer impact, and treat that queue’s depth as a service-level indicator.
- • Verify: reconcile the compensating service’s records against the saga log — every COMPENSATED saga should have exactly one counter-effect per compensatable step, no more and no fewer.
- • Compensation invocation rate, success rate and latency, all separate from forward-step metrics.
- • Count and age of sagas in a compensating state; a non-draining bucket is the primary signal.
- • Duplicate-compensation rate at the compensating service, keyed by compensation identity — the direct measure of whether idempotence is working.
- • Count of compensations that returned "nothing to compensate", which distinguishes healthy tombstones from a broken identity scheme.
- • Business-visible compensation events per day: refunds issued, cancellation emails sent, return labels created. These are the customer-facing cost of the design and belong on a product dashboard, not only an engineering one.
- • Number of sagas escalated because a compensation exhausted its budget — the true count of unrepairable outcomes.
- • Where the business already has reversal semantics — refunds, cancellations, restocking, credit notes — so compensations are real operations rather than engineering fictions.
- • Where the exposure window is short and the intermediate state is not customer-visible, so the "someone will notice" cost is near zero.
- • Where effects are relative and commutative by nature, such as counters and balances, making compensations trivially safe under reordering.
- • Where classifying steps into compensatable/pivot/retriable reveals a better ordering that removes half the compensations entirely.
- • When an irreversible effect sits early in the sequence, so a late failure leaves nothing that can be done.
- • When compensations depend on the same failing dependency that caused the saga to fail in the first place — a correlated failure that guarantees the repair path is broken exactly when needed (Correlated Failure: The Independence Assumption Is Usually False).
- • When the number of compensations grows large enough that they become an untested body of code exercised only in incidents.
- • When the business impact of the visible intermediate state exceeds the cost of the coordination that would have prevented it.
- • When teams reason about compensation as rollback and therefore never design the customer communication that a visible reversal requires.
- • Reorder the saga so irreversible effects come last and the number of compensations drops — cheapest and most effective change available.
- • Use a reservation with an expiry instead of a commit, so the "compensation" is simply not renewing the hold (Leases: Authority With an Expiry Date).
- • Delay the irreversible effect behind a short confirmation window, so most failures happen before anything real occurs — the reason "your order is being prepared" exists.
- • Make the effect naturally reversible by construction: authorise a card rather than capturing it, stage a file rather than publishing it, queue an email rather than sending it.
- • Use two-phase commit for the subset of participants that support it, and compensate only across the boundary that genuinely cannot (Two-Phase Commit: Buying Atomicity With a Promise).
- • Accept the inconsistency and reconcile later, where a rare mismatch is cheaper than the compensation machinery (Reconciliation Is a Component, Not a Cleanup Script).
A refund is not a rollback
| Compensating action | Visible to whom | Can it fail? | Repeat is safe? | |
|---|---|---|---|---|
| Charge cardtypical | Refund — a second ledger entry | Customer, on their statement | Yes — provider 503, expired card | Only with an idempotency key |
| Reserve stocktypical | Release reservation | Every other buyer, as availability | Yes — and it can arrive before the reserve | Yes, if keyed by reservation id |
| Send confirmation emailprotocol | A correction email | The recipient, who already read the first | Yes — bounce, spam folder | No: two corrections is worse than one |
| Dispatch parceltypical | Recall — a physical operation | The courier, the warehouse, the buyer | Yes — often, once it has left | No: two recalls is a second cost |
| Update addressassumption | Write back the remembered value | Anyone reading the record | Yes — and it can clobber a newer write | Only if version-guarded |
What people believe, and what is true
Compensation is just rollback with extra steps.
Rollback is invisible and cannot fail. Compensation is visible, can fail, can duplicate, can arrive out of order, and sometimes does not exist. Every one of those differences generates real work.
We can undo the email by sending a correction.
The correction is a new action. The recipient read the first email; you cannot un-inform someone. The correction changes the recipient’s belief, at best, and doubles the number of emails they received about an order that never existed.
Compensations always run in reverse order.
Reverse order is the common default because dependencies usually run that way. What is actually required is respecting dependencies — independent steps can compensate concurrently, and sometimes must, to bound the exposure window.
Restoring the previous value is a correct compensation.
Only if nothing else changed it in the meantime. Absolute restores are lost updates waiting to happen; use relative operations, or guard the restore with a version check that fails loudly.
If the compensation gets a 404, the original never happened, so we are fine.
The original may be delayed in the network and about to commit. The compensation must leave a tombstone that rejects the late original, or you get a zombie effect nothing will ever clean up.
The compensation only runs in rare failure paths, so it does not need the same rigour as forward steps.
It runs exactly when the system is already degraded, often against the same dependency that just failed. It needs *more* rigour than the happy path, not less.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Undo is not available across services. A refund is a new payment in the other direction: the customer sees a charge and a credit, not an unchanged statement. Design for the trace, not for erasure.
Practical
Give every compensation a deterministic idempotency key derived from the saga id and step name. Retry it forever with backoff and escalate rather than abandon. Make compensating writes relative, never absolute. Make "compensate something that does not exist" a recorded no-op with a tombstone, not a 404. Put emails, notifications and shipments after the pivot, and put the most irreversible one last.
Advanced
Classify every step as compensatable, pivot or retriable, and enforce the ordering that classification implies. A retriable step that can fail for a business reason means the pivot is in the wrong place; move the validation before the pivot and the action after it. Then treat the pivot as the design’s commitment point: before it, availability is preserved by the ability to unwind; after it, availability is preserved only by the guarantee that remaining steps cannot fail permanently. That is the real reason the classification matters — it tells you which steps are allowed to have business-level failure modes at all.
Apply it
- 🔧 Take a saga in your system and classify every step as compensatable, pivot or retriable. Find at least one step that is in the wrong region and propose the reordering.
- 🔧 Implement a compensation that must tolerate arriving before its original. Prove with a test that a delayed original is rejected rather than applied.
- 🔧 Deliberately retry a compensation five times without a key, observe the damage, then add the key and repeat.
- ⚡ A customer received an order confirmation email, a shipping notification, and then a cancellation — in that order — for an order that never had stock. Which step is mis-ordered, and what do you change?
- ⚡ Finance reports duplicate refunds appearing at roughly 0.3% of the compensation rate. What is the most likely cause and how would you confirm it?
- ⚡ A regulator asks why your ledger shows a charge and a refund four seconds apart for a transaction the customer says never happened. Explain what your system did and why the record is correct.
- 💬 Explain why a refund is not a rollback, and name three consequences for the design of a saga.
- 💬 Your compensation for step 2 arrives at the payments service before step 2’s original charge does. What should the payments service do?
- 💬 Which effects in a checkout flow cannot be compensated, and how would you reorder the saga because of it?
- 💬 A compensation has been failing for four hours. Walk me through your response, and say what you would have built beforehand to make it easier.
- 💬 What is a pivot step, and how do you know you have put it in the wrong place?