Resuming a Workflow That Died Halfway Through

"How do we make the agent not crash" is the wrong question. Processes are evicted, containers are rescheduled, deploys roll, model calls hang past the wall-clock limit, and a workflow that spans a human approval will certainly outlive the process that started it. Crashing is normal.

The right question is: what is the smallest durable record from which this workflow can be resumed without repeating a side effect? That question has a precise answer, and it is much less than people expect. You do not need the conversation. You do not need the model’s reasoning. You need the sequence of steps, their identities, their statuses and their results — enough to rebuild a context and to know what has already happened in the world.

This is [[checkpoint-and-log]] applied to a workflow whose plan is generated at runtime. The classical version checkpoints state and logs the operations since the checkpoint; here the operations *are* the state, because the effects are external and cannot be re-derived.

The temptation is to persist the conversation: serialise messages[] after every turn and reload it on resume. It looks like a checkpoint and it is not one, for two reasons that matter.

First, it does not record intentions. If the process died between emitting a tool call and receiving its result, the serialised conversation contains a call with no result — which is exactly the ambiguous state, and the reload gives it back to the model, which will call again. Persisting the conversation faithfully preserves the bug.

Second, it is not addressable. You cannot ask a serialised conversation "did step 4 execute?" without parsing prose. The step log is a table with statuses you can query, sweep, alert on and reconcile against the tool services. That queryability is most of its value in operation.

The right relationship is: the step log is the state; the context is derived from it. On resume you rebuild the context by replaying the log — which also means you can rebuild it differently, summarising old steps or dropping irrelevant ones, without losing any information the workflow depends on.

A step in intended is the interesting case: the effect may or may not have happened. There are exactly four things you can do about it, and choosing consciously per tool is the design work.

Query. If the tool supports a lookup by idempotency key or by a client-supplied reference, ask it. This is the only option that resolves the ambiguity rather than routing around it, and tools should be built or wrapped to support it. Where it exists, use it.

Re-drive idempotently. If the tool deduplicates on the key, call it again: a duplicate collapses into the original and you get the stored result. This is the workhorse and the reason [[agent-idempotency]] comes first in the module.

Compensate. If the effect may have happened and cannot be repeated safely, apply the correcting forward action — refund the charge, cancel the ticket, send a correction. This may be applied unnecessarily if the effect never occurred, so the compensation itself must tolerate that (cancel on a nonexistent booking should be a no-op, not an error).

Escalate. Mark the step abandoned and put it in a human queue. This is not a failure of the design; for genuinely unkeyable, genuinely irreversible effects it is the correct answer, and building the queue is cheaper than pretending the other three options apply.

Once a workflow performs effects across several services, it is a distributed transaction that nobody can roll back, which is precisely the situation [[sagas]] exists for. The mapping is exact: steps are the saga’s local transactions, and each step that must be undoable needs a compensating action.

And the compensation is a new forward action, not an undo — [[compensation-is-not-rollback]] is worth reading before designing any of this. There is no unsend for an email; there is a correction email, or nothing. There is no un-charge; there is a refund, which is a different transaction with its own record, its own fees and its own failure modes. Writing "compensation: reverse the charge" in a design document without naming the actual forward action is how compensations turn out not to exist at implementation time.

Saga design gives one structural instruction that applies directly and is the highest-leverage thing in this lesson: order the steps so the irreversible ones come last. In saga vocabulary this is the pivot — everything before it is compensatable, everything after is retriable-until-success. An agent that drafts, validates, records and then sends has one irreversible step at the end and a trivial recovery story. An agent that sends first and then records has an unrecoverable step at the beginning and no good options.

The plan being generated at runtime does not exempt you from this. It means the *tool definitions* carry the reversibility, and the orchestrator enforces ordering constraints on the plan — refusing, for instance, to execute an irreversible tool while reversible steps remain outstanding, or requiring approval at the pivot.

Checkpointing is not free: every durable write is latency, cost and code. Too fine and every model turn becomes a transaction; too coarse and a crash discards expensive work.

The rule that resolves it: checkpoint at every side-effect boundary, and nowhere else for correctness. The intent-and-outcome pair around each effectful call is mandatory. Everything else — caching model outputs, storing intermediate reasoning, snapshotting a summarised context — is a *cost* optimisation, and should be built and reasoned about separately, because confusing the two leads to systems that persist a great deal and still cannot resume.

And the honest alternative: for a fully read-only workflow, do not resume at all — restart. If no step has an external effect, the entire recovery design collapses into "run it again", which is cheaper to build, cheaper to operate and impossible to get subtly wrong. The cost is repeated model spend, which is often less than the engineering. The moment a single effectful tool enters the workflow, that option closes — which is a good reason to keep the effectful part of a workflow small and at the end.

One more practical constraint: resumption must be bounded. A workflow that resumes, crashes, resumes and crashes forever is a poison message, and the answer is the ordinary one — an attempt counter, and a dead-letter destination after it is exceeded. [[poison-messages]] and [[dead-letter-queues]] apply unchanged, and a workflow with no such bound will eventually consume a queue.