Architectureworkflowstate machinegraphcheckpointingdeterministic

Workflow State Graph

A typed state object moved through a fixed graph of steps with deterministic and conditional edges, capped retries, and checkpoints — the most debuggable way to use LLMs on a task whose shape you already know.

Interview question
Progress

The shape

A workflow is a directed graph. Nodes are steps — some are plain code, some are a single LLM call, occasionally one is a bounded agent. Edges say which step runs next. A single typed state object flows through the graph; every step reads fields it needs and writes fields it produces. There is no growing transcript: the state is the only memory.

The canonical example: START → Classify → Research → Generate → Validate → Approved? → (yes) END / (no) Retry. Classify is a cheap LLM call that writes state.category. Research is code that fills state.sources. Generate writes state.draft. Validate is deterministic code or a judge that writes state.issues. The conditional edge after Validate reads state.issues and state.attempts and decides between END and looping back to Generate.

The LLM decides the content of a step. The graph decides the control flow. That division is what separates a workflow from an agent, and it is the reason workflows are so much easier to test: the set of possible paths is finite and known before you run anything.

Generate-and-validate workflow
yesnoSTARTClassify (small LLM)Research (code / retrieval)Generate (LLM)Validate (schema + checks)Checkpoint stateApproved?Retry (attempts < 3)END
UserLLMAgentToolDataDecisionHumanGuardrail

Typed state, deterministic and conditional edges

Define the state as a schema — a TypedDict, a dataclass, a Zod object — and validate it at every transition. When a step returns a field of the wrong type, the workflow fails at that edge with a precise message, instead of three steps later with a confused model. Typed state is the single most effective debugging tool in this architecture.

Deterministic edges (classify → research) always fire. Conditional edges are pure functions of the state: next = "end" if not state["issues"] else "generate". Keep them free of LLM calls; if a routing decision needs a model, make it a step that writes a field, then route on the field. That way every branch decision is recorded in state and reproducible.

Retries are edges with a counter. attempts lives in state, the conditional edge checks attempts < MAX, and the retry path can feed the validator's issues back into the generate prompt so the second attempt is informed, not a blind re-roll. Cap every cycle; a graph with an uncapped loop is an agent with worse ergonomics.

  • State is one object; steps are (state) -> partial_update; the runner merges updates.
  • Conditional edges are pure functions of state — no I/O, no model calls.
  • Every cycle has a counter in state and a hard cap (2–3 is typical).
  • Retry prompts include the validator's findings; blind retries rarely change the outcome.

A hand-rolled state machine

You do not need a framework for this. A dict for state, a registry mapping step names to functions, a routing table, and a loop is enough for most production workflows, and it stays readable by anyone on the team. Add persistence by writing the state to a store after each step.

A complete workflow runner: dict state, step registry, conditional routing, capped retries, checkpointing.
1from typing import Callable
2 
3State = dict
4Step = Callable[[State], dict]      # returns a partial update
5 
6def classify(s: State) -> dict:
7    return {"category": small_llm(f"Classify into billing|technical|other: {s['question']}")}
8 
9def research(s: State) -> dict:
10    return {"sources": search(s["question"], filters={"category": s["category"]}, k=5)}
11 
12def generate(s: State) -> dict:
13    hint = f"Fix these issues: {s['issues']}" if s.get("issues") else ""
14    return {"draft": llm(GEN_PROMPT, sources=s["sources"], question=s["question"], hint=hint),
15            "attempts": s.get("attempts", 0) + 1}
16 
17def validate(s: State) -> dict:
18    issues = []
19    if len(s["draft"]) > 1200: issues.append("too long")
20    if not cites_sources(s["draft"], s["sources"]): issues.append("uncited claims")
21    return {"issues": issues}
22 
23STEPS: dict[str, Step] = {"classify": classify, "research": research, "generate": generate, "validate": validate}
24 
25def route(step: str, s: State) -> str | None:
26    if step == "classify":  return "research"
27    if step == "research":  return "generate"
28    if step == "generate":  return "validate"
29    if step == "validate":
30        if not s["issues"] or s["attempts"] >= 3: return None      # END
31        return "generate"                                          # retry with feedback
32    raise ValueError(step)
33 
34def run(initial: State, checkpoint: Callable[[str, State], None]) -> State:
35    state, step = dict(initial), "classify"
36    while step is not None:
37        state.update(STEPS[step](state))
38        checkpoint(step, state)          # resume from here after a crash
39        step = route(step, state)
40    return state

Checkpointing and resumption

Because the state is explicit and serialisable, you can persist it after every step. A crash or deploy mid-run resumes from the last checkpoint instead of restarting — and re-running a step is safe only if that step is idempotent, which is why Idempotency matters for the tool calls inside steps.

Checkpoints double as human-in-the-loop gates. A step can end the run with status = "awaiting_approval"; a human edits the state (approves, changes a field), and the runner continues from the next edge. That is the mechanism behind Approval Gates and Risk Classes in most workflow frameworks.

Checkpoints are also your regression test fixtures. Save the state before generate from a real run, and you have a deterministic input for evaluating prompt changes to that one step in isolation.

Why workflows are more debuggable than free agents

A free agent's behaviour is a function of the whole transcript; to reproduce a bug you replay every turn and hope the model makes the same choices. A workflow's behaviour is a function of the state at one node; to reproduce a bug you load the checkpoint and re-run one step. The search space for "where did it go wrong" shrinks from every step to one node.

The set of paths is enumerable. You can unit-test each step with fixture state, property-test the routing function, and measure each LLM step against its own golden set as in Deterministic Evaluators. Latency and cost are predictable: sum over steps plus a bounded number of retries — you can put the p95 in a spreadsheet before shipping.

The cost is rigidity. If the task turns out to need a step you did not draw, the workflow cannot invent it. That is a feature until the request distribution is genuinely open-ended, at which point a bounded agent as one node, or a Supervisor Architecture, is the escalation.

  • Complexity: 2–3 — a graph and typed state, but no coordination policy.
  • Latency: 2 — fixed number of hops; parallel branches where independent.
  • Cost: 2 — no growing transcript; each step sees only the fields it needs.
  • Reliability: 5 — bounded paths, capped retries, validated state.
  • Debuggability: 5 — checkpoint-and-replay of a single node.

Key points

  • Workflow = typed state + fixed graph of steps + deterministic and conditional edges. LLMs decide content; the graph decides control flow.
  • Conditional edges are pure functions of state; model-driven routing writes a field first and routes on it.
  • Every cycle has a counter in state and a hard cap; retries should carry the validator's feedback.
  • Checkpoint the state after each step for resumption, approval gates, and regression fixtures.
  • Bugs reproduce by loading one checkpoint and re-running one step — far cheaper than replaying an agent transcript.
  • Cost and latency are computable in advance; the price is rigidity when the task shape changes.

When to use — and when not to

Use it when
  • The task decomposition is known: classify, fetch, generate, validate, deliver.
  • You need predictable latency and cost, or an SLA.
  • Human approval must happen at a specific point in the process.
  • Compliance or audit requires the exact sequence of operations to be explainable.
  • Long-running jobs that must survive restarts.
Avoid it when
  • The steps are genuinely unknown until the model sees the input — an agent node or a supervisor is required.
  • The workflow has so many conditional branches that the graph is a hand-drawn agent; simplify or accept an agent.
  • A single LLM call with structured output already solves it — a graph of one node is ceremony.
  • Interactive conversations where the user changes the goal mid-run and the graph cannot follow.

Failure modes

  • Uncapped retry loop: validate always fails, generate always retries, cost climbs until the provider rate-limits you.
  • State schema drift: a step writes sources as a string one day and a list the next; downstream steps silently misbehave without validation.
  • Non-idempotent step re-executed after a resume sends the customer two emails.
  • Routing logic that calls the model inside the edge makes branch decisions unreproducible.
  • Graph sprawl: 40 nodes and 60 edges that nobody can hold in their head — an agent in disguise.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

Best reliability and debuggability of any LLM architecture — when the task shape is fixed.

← Supervisor ArchitectureRouter Architecture →