Where the Probabilistic System Ends
Four boundaries — workflow, permission, output and fallback — and the argument that each one must be a thing in the code rather than an understanding in someone's head.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
Where exactly does my deterministic system stop and the probabilistic one begin, and can a new engineer see that line without being told?
Six months in, the assistant works. Nobody on the team can answer, on a whiteboard, which parts of it are guaranteed and which are likely — and two engineers give different answers.
The boundary is obvious — it is where we call the model. Everyone on the team knows which parts are AI. This is true on the day the system is written, by the people who wrote it, and it is the reason no boundary gets built.
A new engineer adds a feature that reads the model's free-text summary and branches on whether it contains the word "urgent". The probabilistic part just grew, and no review caught it, because nothing in the code said where the line was (Invariant Leaks).
- A new engineer adds a feature that reads the model's free-text summary and branches on whether it contains the word "urgent". The probabilistic part just grew, and no review caught it, because nothing in the code said where the line was (Invariant Leaks).
- A tool is added by another team with the run's full privileges, because that is what every existing tool had. The permission boundary was a convention, and conventions do not survive contributors (Least Privilege as a Design Decision).
- The run has no terminal state, so a failure looks like a slow success and the UI waits. "Did it finish?" turns out to be unanswerable (State Machines).
- Retry means re-running the whole run, which re-runs the tools that already succeeded, because nothing recorded which steps had completed and which had effects (Resuming a Workflow That Died Halfway Through in Distributed Systems).
- Behaviour degrades and the team cannot tell whether it was a prompt edit, a model change or a code deploy, because the three are recorded in three places with no shared identifier (Stable Identifiers).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The system already exists and is in production, so every boundary has to be introduced incrementally rather than designed in (Incremental Migration).
- Three teams contribute tools, and none of them sees the whole run.
- Behaviour changes arrive from outside the deploy pipeline — a prompt edit, a provider-side model change — so "we did not deploy" is not evidence that nothing changed (Deploys Are the First Suspect in Backend Engineering).
- Every run terminates: a step budget, a cost budget and a wall-clock deadline, all enforced by the runtime rather than by instruction (Budgets, Limits and Termination in the agentic domain).
- Every effect on your systems has a deterministic authorization decision behind it, made against the acting user (Least Privilege as a Design Decision).
- Every run ends in a state a caller can branch on — succeeded, escalated, refused, failed — never in "here is some text, work it out" (Explicit State).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The runtime owns the workflow boundary: which steps may run, in what order, how many, and when the run stops. Termination is a runtime property and never an instruction to the model.
- The tool layer owns the permission boundary, per tool, per object, against the acting user (Designing a Tool Interface).
- A parse step owns the output boundary: model output becomes a typed value or a typed failure, and nothing downstream ever handles a raw string (Result Types).
- The application owns the fallback boundary: what the product does when the probabilistic part produces nothing usable. This is a product decision expressed in code, not an error handler (Designing for Failure).
- Workflow: the set of reachable steps is a value in the code — a graph or a state machine — not an emergent consequence of what the model decides to call (Workflow State Graph in the agentic domain).
- Permission: what a run may do is granted per run and per tool, so the answer to "what could this run have done" is a list rather than an argument (Capability Passing).
- Output: the last place a string exists is the parser. After it, the system holds domain types, and a type error becomes possible where before there was only disappointment (Parse, Do Not Validate in Backend Engineering).
- Fallback: the degraded path is a first-class branch with its own tests and its own UI, because it runs every day (Error Boundaries).
Four boundaries, and what each one is made of
The word "boundary" gets used loosely enough to mean nothing. Made concrete, there are four here, and each one is a specific artifact you can point at in a pull request — which is the test of whether it exists.
They are separable and teams usually have some of them. Output boundaries are the most commonly built, because the pain arrives fastest; permission boundaries are the most commonly assumed, because nothing goes wrong until something goes very wrong.
- Workflow — a state machine with a terminal state and budgets the runtime enforces. Not "the prompt says to stop after five steps" (Workflow State Graph in the agentic domain).
- Permission — capabilities granted per run, per tool, per object, checked against the acting user (Capability Passing, Least Privilege as a Design Decision).
- Output — one parse step; after it, no component in your system handles model text (Parse, Do Not Validate in Backend Engineering).
- Fallback — a designed degraded product, tested and staffed, because it runs daily rather than during outages (Graceful Degradation: Which Dependency Is Actually Critical in Distributed Systems).
The run as a state machine
The forbidden transitions are the part worth arguing over, and they are the part usually left implicit. Each one below corresponds to an incident that teams have actually had, and each should be a failing test rather than a shared understanding.
Note that awaiting_human is a real state and not an error path. In most agent products it carries a large fraction of traffic, and modelling it as an exception is how a support queue ends up invisible to the people who staff it.
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| planning | model proposes a call | awaiting_tool | step budget not exhausted && tool is in this run's grant | decrement step budget |
| awaiting_tool | validated and authorized | executing | schema valid && acting user authorized for this object | record idempotency key |
| awaiting_tool | above value threshold or below confidence threshold | awaiting_human | a deterministic threshold in code, not a model judgement | — |
| awaiting_tool | policy denies | refused | — | record the rule that denied it |
| executing | tool returned | planning | — | append typed result; effect is now durable |
| planning | model signals completion | succeeded | a typed final result parses | — |
| planning | step, token or time budget exhausted | failed | — | retain partial results |
| awaiting_human | human approves | executing | approver authorized independently of the run | — |
| awaiting_human | human takes over | escalated | — | — |
- planning → executing — It skips parse, validation and authorization — the entire output and permission boundary. This is what "the model called the function directly" means, and it is the default in most quick implementations.
- awaiting_human → succeeded — A run cannot approve itself. If the model can move the run past a human gate, the gate is decorative and the threshold that created it means nothing (Approval Gates and Risk Classes in the agentic domain).
- executing → executing — Re-entering execution for the same step is a double effect. A retry must re-enter through
awaiting_toolso the idempotency key is checked (Idempotency by Design). - failed → planning — Resuming a budget-exhausted run silently defeats the budget. Resumption is a new run with a new budget and an explicit decision to spend more (Budgets, Limits and Termination in the agentic domain).
- refused → planning — A refusal that the run can retry its way past is not a refusal. This is the transition that appears when someone adds "if denied, try a different approach" to a prompt.
The three-way split at the end — refused, failed, escalated — is deliberate and is the part teams most often collapse into one "error". They demand different responses: refused is correct behaviour and should be explained to the user, failed is an operational problem and should page someone, escalated is a queue that needs staffing (An Error Taxonomy That Survives Contact).
Introducing the boundaries into a system that already works
None of this can be done as a rewrite, and a rewrite is what teams reach for once they realise the boundaries are missing. The order below is chosen so each step is independently valuable and independently shippable, and so the two that reduce risk arrive first (The Strangler Pattern).
The step most often skipped is the first, and skipping it is why these projects stall: without a written list of which decisions are currently probabilistic, the work has no definition of done and no way to show progress.
- 1Inventory the decisions
List every decision the run makes and mark each guaranteed or likely. Two engineers doing this separately and disagreeing is the finding.
fails by Being skipped because "we know what the AI does" — which is the belief the exercise exists to test.
- 2Parse at the edge
Introduce a typed boundary on model output. Cheapest step, largest immediate reduction in weird bugs, and no behaviour change.
fails by Keeping a
rawfield for debugging, which three features start reading within a quarter (Exposing Too Much). - 3Grant capabilities per run
Replace the ambient tool set with an explicit per-run grant. Start by granting exactly what is used today, then narrow.
fails by Narrowing before measuring, which breaks working flows and gets the whole effort reverted (Characterization Tests).
- 4Make the run a state machine
Give the run explicit states and one terminal state, then enforce budgets in the runtime. This is where termination stops being a request.
fails by Modelling the happy path only, so
awaiting_humanandrefusedremain error handlers (Designing the Happy Path Last). - 5Design the fallback
Decide what the product does when there is no usable answer, build it, and test it as a normal path.
fails by Building it and never exercising it, so it fails the first time it matters (Failure-Aware Feature Design).
- 6Attribute behaviour
Record model id, prompt revision and run id on every step so a change can be traced to one of the three things that could have caused it.
fails by Recording them in three systems with no shared id, which is the same as not recording them (Correlation IDs: Turning Lines Into a Story in Backend Engineering).
Each step is shippable alone and each leaves the system better, which is what makes this a migration rather than a rewrite. If a step cannot be shipped independently, it has been scoped wrong (Incremental Migration).
How to build it
Most important first.
- Write the run as an explicit state machine with a terminal state, and make every transition a thing the runtime performs. A run that can only end by the model choosing to stop has no termination guarantee at all (State Machines).
- Enforce budgets in the runtime: steps, tokens, wall clock, money. Each has a defined terminal state, and exceeding one is an ordinary outcome rather than a crash (Budgets, Deadlines and Step Limits in Backend Engineering).
- Grant capabilities per run rather than per service. A run initiated from a support conversation gets refund tools; a run summarising a changelog gets none (Capability Passing).
- Parse at the edge and make the boundary visible in the type names.
ModelTextandRefundIntentshould not be interchangeable, and the compiler is a better boundary guard than a code review comment (Making Illegal States Unrepresentable). - Record the model id, the prompt revision and the run id on every step, so a behaviour change can be attributed to one of the three things that could have caused it (Debuggability by Design).
- Make each step's effect recoverable: record what completed, so a resumed run skips what already happened rather than repeating it (Idempotency by Design).
- Design the fallback as a product, not as a catch. "We could not draft a reply — here is the ticket and the three most similar past ones" is a feature; a spinner that never resolves is what happens when the fallback is an afterthought (Graceful Degradation: Which Dependency Is Actually Critical in Distributed Systems).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Adding a step: one node in the state machine, one tool grant, one test. Bounded, because the graph is a value someone can read rather than a behaviour that emerges from a prompt.
- A model upgrade: the boundaries do not move at all, which is the point. What changes is the *distribution* inside the probabilistic region — parse failure rates, escalation rates, tool-selection accuracy — and because those are metrics at named boundaries, the upgrade is a measured comparison rather than a hunt. The upgrade also reliably changes how often the fallback path runs, which is why the fallback needs capacity planning and not just correctness.
- Tightening a permission: cheap and therefore actually done. Because grants are per run, narrowing one is a config change with a blast radius you can name in advance (Blast Radius: If This Is Wrong, How Much Does It Affect? in DevOps).
- What stays expensive: moving a decision across the line. Promoting something from deterministic to model-decided, or demoting it back, touches the state machine, the types, the tests and the evals — deliberately. That change should be expensive enough to be noticed, because it is a change to what the system guarantees.
- Explicit boundaries cost flexibility. A run that could have improvised a fifth step now cannot, and some of those improvisations would have been correct and useful.
- Four boundaries is real machinery — a state machine, a grant model, a parser, a fallback path — for a feature that worked without any of it. For a low-stakes internal tool, that is over-design and the honest answer is to skip most of it (Over-Design and Under-Design).
- Per-run grants mean a wiring layer that has to know which run type gets which tools, and that layer becomes a place teams have to coordinate (Wiring and the Composition Root).
What can go wrong
- The boundary is documented and not enforced, so it holds until the first deadline. A boundary that is a paragraph in a design doc is a boundary that already does not exist (Documentation Decay).
- Budgets exist but have no terminal state, so exceeding one throws from deep inside the runtime and the caller sees a generic error instead of "escalated: budget exhausted".
- The fallback path exists and is never exercised, so the first time it runs in anger it fails too — the classic failure of a mitigation nobody tests (Failure-Aware Feature Design).
- Typed outputs are introduced and one
raw: stringfield is kept "for debugging". Within a quarter, three features read it, and the output boundary is gone (Exposing Too Much).
- The runtime depends on the tool registry and the budget policy; tools depend on domain services; domain services depend on nothing agentic. Every dependency points away from the probabilistic side (Dependency Direction).
- The application depends on a run's terminal state, not on its transcript. Depending on the transcript is how a UI ends up parsing prose (Designing a Module Interface).
- "Boundaries mean the model does less." They mean what the model does is *known*. A well-bounded system can delegate more, not less, because the consequences of being wrong are contained (Least Privilege as a Design Decision).
- "The state machine is the agent framework's job." A framework can supply the mechanism and cannot supply the decision about which states your product has. Delegating the second to a framework is how teams end up with a run lifecycle they did not choose (What a Framework Charges).
- "We will add the fallback when we see failures." You are seeing them; they are being rendered as confident output. The fallback is not for outages, it is for the several times a day when the answer is unusable (Partial Failure).
- "Typed outputs solve non-determinism." They convert an unbounded output space into a bounded one, which is a large improvement and not a guarantee. A well-typed
RefundIntentcan still name the wrong order (Structured Outputs in the agentic domain).
- exposing-too-much
- boolean-flag-explosion
Testing it, and how it ages
- A test per forbidden transition. The state machine's
forbiddenlist is not documentation — each entry should be a test that fails if the transition becomes reachable (Invalid Transitions). - Budget tests: a run that would exceed the step, token or time budget terminates in the right state, with the right partial results retained.
- Fallback tests as first-class cases: model unavailable, output unparseable, confidence below threshold. Each has an asserted product behaviour, not just an absence of a crash.
- A permission test per run type asserting that tools outside the grant are not merely unused but unreachable (Least Privilege as a Design Decision).
- The line moves toward the model as capability improves, and the well-designed version of this is a reviewed change with an eval and a rollback, not a drift nobody noticed.
- The line moves the other way after every serious incident, and a design where that is a small refactor is worth a great deal at exactly the moment it is needed.
- Permission grants accumulate and are never revoked unless someone owns revoking them. Usage-driven pruning is the maintenance task this design creates (Access Review in DevOps).
- Eventually the state machine outgrows being a switch statement and wants its own representation. That is success; the failure mode is letting it stay implicit until nobody can draw it (Explicit State).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat an unenforced boundary erodes under contributors and deadlines is a property of teams, not of models, and it is why each of the four is specified here as a code artifact rather than as a practice.
- SCALE-SPECIFICAt one team and one tool, the boundaries are genuinely in everyone's head and the machinery is waste. They start paying at the point where someone who did not build the system contributes a tool — which in practice is around the second team, not the second engineer.
- LIFETIME-SPECIFICAgent runtimes, protocol layers and orchestration frameworks are changing fast enough that any specific mechanism here will look dated quickly. The four boundaries are the durable part; where they are implemented — your code, a framework, a platform — is not, and should be a decision you can revisit (Reversible and Irreversible Decisions).
- CONTESTEDThe strongest counter-argument is that explicit workflow graphs recreate the rigidity agents were adopted to escape: if you can enumerate the states, you did not need an agent, and a team that graphs everything ends up with a worse workflow engine plus a model bill. That is a fair description of a real failure mode. The reconciliation most practitioners reach is that the *steps* can be open-ended while the *bounds* — termination, permission, output shape, fallback — are not; the disagreement is about how much open-endedness inside those bounds is useful, and that is genuinely unsettled.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — the fallback path is a reliability design with a capacity implication, and deciding how much degraded capacity to staff for is their question.
- — System Design — run state that must survive a process restart turns the state machine into a persisted workflow, and the durability and recovery mechanics belong there.