The question this answers
Who owns the knowledge of what step a saga is on — and what changes when the answer is "one component"?
The saga’s position is a single durable state machine: at any moment exactly one component holds the authoritative answer to "which steps have completed", provided the orchestrator persists each transition before dispatching the corresponding command. It does not guarantee atomicity, isolation, or that participants agree with the orchestrator at any given instant — a participant may have committed a step the orchestrator has not yet recorded.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The orchestrator knows what it dispatched and what replies came back. It does not know what participants actually did — a dispatched command whose reply was lost leaves the orchestrator believing a step is incomplete when it has committed. Participants know even less: each knows only its own local transaction and has no idea what saga it belongs to unless the orchestrator tells it. The authoritative state is the orchestrator’s *record of its beliefs*, which is not the same thing as the world.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The shape: commands out, replies in
An orchestrator is a state machine with a durable store. It reads its current state, decides the next command, persists the decision, sends the command, and waits. When a reply arrives it persists that, transitions, and repeats. Participants are dumb in the useful sense: they expose operations and know nothing about the workflow.
The immediate benefit is that the workflow exists as an artefact. It can be read as code or a diagram, versioned, tested end to end, and queried at runtime. "Why is this order stuck?" has a database row as its answer rather than an exercise in log archaeology across six services. For any workflow with a business owner and an SLA, that property alone usually decides the choice.
The second benefit is that failure handling has an owner. Compensations, retries, timeouts per step, escalation to a human — all of it lives in one place, written once, rather than being reimplemented in each participant’s event handler. That is a large reduction in the amount of rarely-exercised code in the system.
Persist before dispatch — and why the alternative is worse
The orchestrator has two actions per step: write its state, and send a command. They are not atomic with each other, so a crash can land between them, and the *order* determines which failure you get.
Persist then send: a crash after persisting means the orchestrator recovers believing it dispatched a command it may not have sent. It re-sends. The participant may therefore receive the command twice — at-least-once, which idempotent participants absorb. Send then persist: a crash between them means the orchestrator recovers with no record of the command. It never re-sends, the reply arrives for a step the orchestrator does not know about, and the saga is stuck with committed work nobody is tracking. That is unrecoverable without human intervention.
So persist-then-send is the only safe order, and it forces at-least-once command delivery, which forces idempotent participants. The same argument produced the outbox in Atomicity Stops at the Process Boundary and the saga log in Sagas: Trading Isolation for Availability; it is one rule appearing in three places. Note also that the orchestrator’s own state write and its command send are themselves a dual write, which is why mature orchestrators dispatch through an outbox rather than calling participants directly.
1# SAFE — persist, then send2persist(saga, step=3, status=DISPATCHED, key=k3)3send(inventory, ReserveStock, key=k3) # crash here → resume re-sends4# recovery: step 3 is DISPATCHED, outcome unknown → re-send with key k35# participant dedups on k3. Worst case: a duplicate command, absorbed.6 7# UNSAFE — send, then persist8send(inventory, ReserveStock, key=k3) # crash here → nothing recorded9persist(saga, step=3, status=DISPATCHED, key=k3)10# recovery: orchestrator believes step 3 never started.11# Stock is reserved. Nobody will ever release it. Requires a human.What you have built: a dependency every workflow shares
The orchestrator is now on the critical path of every business operation it owns. If it is down, no saga advances — and here is the operationally nasty part: every participant is healthy. Dashboards are green, error rates are zero, latency is fine, and orders stop moving. The failure has no symptom in any of the places people habitually look.
It is also a state store with a particular access pattern: many long-lived rows, each read and written a handful of times over minutes to days. That store’s availability is your workflow’s availability, and its write latency is on every step boundary. Sagas that live for days accumulate; a workflow that runs for a week at ten thousand starts a day means seventy thousand live instances, all of which must be resumable.
And it centralises change. Every workflow modification is a deploy of one component, which is convenient for reasoning and inconvenient for teams: the orchestrator becomes a place where several teams’ logic meets and where their release cadences collide. This is the coupling that choreography advocates are actually objecting to, and the objection is real.
- Availability — orchestrator down means no progress anywhere, with no error signal at any participant.
- Scaling — sagas must be partitioned across orchestrator instances, usually by saga id, with the same rebalancing problems as any partitioned system (Rebalancing: A Load Spike You Schedule for Yourself).
- Hot instances — a workflow with a very long tail keeps state alive for weeks; storage and scan cost grow with the tail, not the throughput.
- Team coupling — one repository, one deploy, several teams’ business rules.
- Blast radius — a bug in the orchestrator affects every workflow it runs, not one step of one workflow.
The version-migration problem nobody plans for
This is the operational issue that separates teams who have run an orchestrator from teams who have read about one. At any moment there are thousands of sagas mid-flight. You deploy a change that inserts a new step between steps 3 and 4. Every in-flight saga currently at step 3 will now resume into a workflow definition that does not match the one it started under.
The failure shapes vary and all are unpleasant: a saga resumes at a step index that now means something else; a saga is asked to compensate a step the new code no longer knows how to compensate; a saga’s persisted state contains a field the new state machine cannot deserialise. The symptom is usually a burst of stuck sagas immediately after a deploy, followed by a rollback that leaves sagas that already advanced under the new code stranded in the old one.
The standard answers are to version the workflow definition and pin each saga instance to the version it started under, keeping old definitions deployed until the last instance drains; or to make every state transition data-driven and additive so old instances remain valid. Both cost real work. The point for a designer is that an orchestrator turns "deploy a code change" into "migrate a population of running processes", and that must be budgeted for at the start, not discovered at the first breaking change.
$ workflow versions --name checkout VERSION DEPLOYED RUNNING OLDEST RUNNING v7 2026-08-25T09:00:00Z 3,204 00:04:12 v6 2026-08-18T09:00:00Z 881 6d 22:31:07 <- cannot delete yet v5 2026-08-04T09:00:00Z 3 21d 03:55:41 <- investigate these # v5 has three instances stuck for three weeks. Deleting v5's definition # would make them unresumable. They are why the "just delete old code" # instinct is wrong for an orchestrator.
When orchestration is the right call
Choose orchestration when the workflow itself is a business asset: it has an owner, an SLA, a support process, and people who ask where a given instance is. Order fulfilment, loan origination, onboarding, provisioning, refund processing — these are workflows that someone is accountable for end to end, and giving that accountability a home in the code is worth the coupling.
Choose it also when the failure handling is complex — many compensations, per-step timeouts, retry budgets, human approval gates. Concentrating that logic is a large reduction in total system complexity even though it is an increase in one component’s complexity.
It is the wrong call when the "workflow" is really just a fact that several services independently care about, when the steps are genuinely optional, and when the coupling to a shared component would slow teams down more than the visibility helps. That is Choreography: The Workflow Nobody Wrote Down’s territory, and the comparison is honest in both directions.
| Axis | Orchestration | Choreography |
|---|---|---|
| Where the workflow livesprotocol | One state machine you can read | Emergent from subscriptions |
| "Where is order 4821?"typical | A row lookup | Join several event logs |
| Couplingtypical | Participants coupled to orchestrator | Publishers coupled to event schema |
| Adding a steptypical | Change one component; migrate in-flight instances | Deploy a new subscriber; nobody else changes |
| Failure handlingtypical | Centralised, written once | Distributed across handlers |
| Single point of stallassumption | Yes — the orchestrator | No — but the broker still is |
| Cyclic dependency risktypical | Low — flow is explicit | High — chains form silently |
Key points
- The orchestrator holds the workflow as an explicit, durable, queryable state machine; participants know nothing about it.
- Persist the transition before dispatching the command — the reverse order loses commands unrecoverably.
- That ordering forces at-least-once command delivery, which forces idempotent participants.
- An orchestrator outage stops every workflow while every participant reports perfect health.
- Deploys become migrations: in-flight instances must be versioned and pinned, and old definitions kept until they drain.
- The right choice when a workflow has a business owner, an SLA and complex failure handling.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The orchestrator loads the saga’s durable state, or creates it on first request.
- • It computes the next command from the state machine definition and the accumulated results.
- • It persists the intended transition, including the idempotency key it will use for the command.
- • It dispatches the command — ideally through an outbox, so its own state write and the send are atomic.
- • The participant executes a local transaction and replies; the reply is persisted before the state advances.
- • On failure, the orchestrator switches to the compensating path and drives it with the same persist-then-dispatch discipline.
- • The saga reaches a terminal state, and its record is retained for querying and reconciliation.
- • The orchestrator crashes between persisting and dispatching, so the command is re-sent on resume.
- • A reply is lost, so the orchestrator re-dispatches a command the participant already executed.
- • The orchestrator’s state store is unavailable, so no saga can advance in either direction.
- • A deploy changes the workflow definition under in-flight instances.
- • The orchestrator partition assignment changes and two instances briefly drive the same saga.
- • A participant is unreachable for longer than the step timeout, and the orchestrator must decide between waiting and compensating under A Timeout Tells You Nothing About Whether It Happened.
- • Silent global stall: the orchestrator’s state store hits a connection limit. Every participant service reports 100% availability and normal latency; orders simply stop moving. Nothing in the standard dashboard shows a problem.
- • Post-deploy stuck burst: sagas in a particular step spike into a stuck state within minutes of a release, because their persisted state no longer matches the deployed workflow definition. Rolling back strands the instances that already advanced.
- • Duplicate commands after resume: the operator sees participants reporting a rise in idempotent-replay hits immediately after every orchestrator restart. This is healthy — but only if the participants dedup; if one does not, the same restart produces duplicate business effects.
- • Split driving: two orchestrator instances both believe they own a saga after a rebalance and both dispatch step 4. The participant sees two commands with the same key (harmless) or with different keys (a duplicate effect).
- • Ageing tail: a small number of sagas never terminate and accumulate for months, keeping old workflow versions undeletable. The operator sees storage growth and a version table that will not shrink.
- • All coordination is concentrated at the orchestrator: it is the only component that must agree with itself, which is why the design is easy to reason about.
- • Each step boundary costs a durable write to the orchestrator’s store, so the store’s write latency is multiplied by the number of steps in every saga’s total duration.
- • Ownership of a saga instance must be exclusive during a dispatch, which is a leadership or partition-ownership problem — the same one Distributed Locks: What They Are Actually For and Leases: Authority With an Expiry Date describe.
- • Availability is coupled: every workflow shares the orchestrator’s fate, which concentrates blast radius in exchange for concentrating understanding.
- • Committed participant work is durable and unaffected by orchestrator failure; only the driving stops.
- • The orchestrator’s record of saga state is durable and resumable, so no in-flight saga is lost provided persist-before-dispatch was honoured.
- • The orchestrator’s view may lag the world: a step may have committed at a participant while the orchestrator still shows it as dispatched.
- • No isolation is added; the saga’s intermediate states remain visible exactly as in any saga.
- • Detect: alert on orchestrator state-store health and on saga throughput (transitions per minute), because throughput hitting zero is the only symptom of a total stall.
- • Contain: shed new saga starts before degrading in-flight ones, so the backlog of resumable work stays bounded.
- • Recover: resume from persisted state, re-dispatching any step whose outcome is unknown, relying on participant idempotence.
- • Reconcile: after any orchestrator incident, diff the orchestrator’s view of each saga against each participant’s records for the affected window — the two can legitimately disagree, and you need to know where.
- • Verify: confirm the stuck-state buckets drain and that no workflow version has instances older than its expected maximum duration.
- • Saga transitions per minute — the liveness signal. Zero with healthy participants is the orchestrator-stall signature.
- • Distribution of sagas by state and by workflow version, with an age histogram per bucket.
- • Orchestrator state-store write latency, which sits on every step boundary.
- • Command re-dispatch rate, which rises after every restart and should return to baseline.
- • Oldest running instance per workflow version, which tells you when an old definition can safely be deleted.
- • Count of sagas whose orchestrator state disagrees with the participant’s record, produced by the reconciliation job rather than by the orchestrator itself.
- • Workflows with a business owner, an SLA, and support staff who must answer "where is this order?".
- • Complex failure handling — many compensations, per-step timeouts, approval gates — that would otherwise be duplicated across services.
- • Long-running processes measured in hours or days, where in-memory driving is not an option and durable state is required anyway.
- • Regulated processes that must produce an auditable record of exactly which steps ran, when, and in what order.
- • When onboarding new engineers: an explicit state machine is dramatically cheaper to learn than an emergent one.
- • When the "workflow" is two steps and a notification — the orchestrator is more machinery than the problem justifies.
- • When many teams must change the same workflow definition frequently, so the orchestrator becomes a release bottleneck.
- • When workflows change often and in-flight instances are long-lived, making every deploy a migration exercise.
- • When it becomes a place to put business logic that belongs in the participants, gradually turning into the distributed monolith it was meant to avoid (The Distributed Monolith: All of the Cost, None of the Autonomy).
- • Choreography, where services react to events and no component owns the flow (Choreography: The Workflow Nobody Wrote Down).
- • A durable workflow engine rather than a hand-rolled orchestrator, so persistence, retries, versioning and resume are solved for you.
- • Collapse the workflow into one service where the steps genuinely belong to one owner (Atomicity Stops at the Process Boundary).
- • Client-driven orchestration for short flows, where the caller drives the steps and holds the state — cheap, but it loses durability the moment the client goes away.
- • A hybrid: orchestrate the core transactional path, and let peripheral reactions (analytics, notifications, search indexing) be choreographed off its events.
One component owns the workflow — and one log entry decides whether that survives a restart
What people believe, and what is true
The orchestrator knows the true state of the saga.
It knows what it dispatched and what replies it received. A committed step whose reply was lost is invisible to it. The orchestrator holds authoritative *intent*, not authoritative *outcome*.
Orchestration reintroduces the monolith.
It centralises the workflow, not the business logic. The distinction matters: participants still own their data and their rules. It becomes a monolith only if you let step logic migrate into the orchestrator.
The orchestrator is a single point of failure, so choreography is more available.
Choreography still depends on the broker, which is equally central. You move the single point rather than removing it — and the broker cannot tell you where order 4821 is.
We can deploy workflow changes like any other code change.
There are thousands of running instances holding state that matches the old definition. Without versioning and pinning, a deploy strands them.
A saga library gives us orchestration for free.
It gives you persistence and retry. Versioning in-flight instances, deciding compensations, and choosing step order remain entirely yours.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One component holds the workflow as a state machine, sends commands, and records replies. You can ask it where any instance is. Everything now depends on it.
Practical
Persist every transition before dispatching. Give each dispatch a deterministic idempotency key so resume is safe. Expose transitions-per-minute as your liveness metric, because a total stall has no other symptom. Version workflow definitions and pin instances, and track the oldest running instance per version so you know when old code can go.
Advanced
The orchestrator’s state write and its outbound command are a dual write, so a mature implementation dispatches through an outbox and treats command delivery as at-least-once. Saga ownership must be exclusive during dispatch, which makes instance-to-worker assignment a partitioning problem with all the usual rebalancing hazards: during a reassignment two workers may briefly drive one saga, and only participant idempotence keeps that harmless. Design the key so that both workers derive the same one.
Apply it
- 🔧 Build a three-step orchestrator, then kill it at every point in the cycle and verify that resume produces the same final state each time.
- 🔧 Introduce a workflow version change while instances are in flight and observe what breaks. Then implement version pinning and repeat.
- ⚡ All services report green, but no orders have progressed in twenty minutes. Where do you look first and what metric would have told you immediately?
- ⚡ After a rebalance, one participant reports receiving the same command from two different orchestrator instances. Is this a bug? What determines the answer?
- 💬 Why must an orchestrator persist its state before sending a command, and what specifically goes wrong in the other order?
- 💬 The orchestrator is down. What does your monitoring show, and why is that dangerous?
- 💬 You need to insert a new step into a workflow with 40,000 in-flight instances. Walk me through the deploy.
- 💬 When would you choose choreography over orchestration for the same workflow?