The Failures That Produce No Errors

Distributed debugging leans on replay: capture the inputs, re-run, observe. [[distributed-debugging]] and deterministic-replay in Concurrency both depend on the assumption that the same inputs produce the same execution. A model breaks that assumption at the first step.

Re-running an agent with the same prompt may produce a different plan, call different tools, in a different order, and reach a different answer — even at temperature zero, since batching, model versions, provider-side routing and floating-point non-associativity all move the result. So "I reproduced it locally" is a much weaker statement here than elsewhere, and a bug that appears in one run in fifty is genuinely hard to pin down.

The workable substitute: record outputs, not just inputs. If every model response, tool call and tool result is captured, a run can be replayed *deterministically from the recording* — feed the stored responses back instead of calling the provider. That gives you a reproducible trace of what actually happened, which is what debugging needs, at the cost of storing a lot of text. It does not let you ask "what would the model do differently", but that is a different question and usually not the one blocking you.

This also changes what a test is. An assertion that the agent produces a specific output is flaky by construction. An assertion that the agent, given a recorded trace, took a permitted sequence of actions is stable. Structure and constraint are testable; content is evaluated statistically, which is what evals-overview is for.

This one quietly invalidates an assumption the rest of the domain relies on. When an ordinary RPC is retried, the retry means the same thing: same method, same arguments, same intended effect. That is why retry policies are safe to configure generically.

When a model call is retried, the provider regenerates. The new response may call a different tool, call the same tool with different arguments, or decide no tool is needed. The retry is a fresh non-deterministic sample, not a repetition of a request. Which means retrying at the model-call layer is not equivalent to retrying at the step layer, and conflating them produces a specific bug: the first attempt called create_ticket, the retry calls escalate_to_human, and now the workflow has done both.

The consequence for design is to be explicit about which layer retries. Retrying the *transport* — a connection reset before any tokens arrived — is safe and should be automatic. Retrying the *generation* is a decision to allow a different path, and should be bounded and recorded. Retrying the *step* means re-driving the tool with the same idempotency key, which is what [[agent-idempotency]] provides and which is the layer where retry is actually safe.

The nastiest variant is a partial stream. The connection breaks after the model has emitted half a tool-call block. You have a syntactically incomplete instruction: a tool name, some arguments, no closing structure. It cannot be executed, and it cannot be assumed not to have been intended. The only safe handling is to treat the whole turn as not having produced a call, and to make sure the *previous* turn’s effects are already durably recorded so that re-generating cannot repeat them.

The context window is a bounded, lossy replica of the workflow state, and it degrades in ways with no error signal at all. Truncation drops the oldest turns. Summarisation compresses ten steps into three sentences and loses the tool result that mattered. A new process rebuilds context from a log that was missing a step. A subagent is given a scoped context that omits what its parent already tried.

In every case the model’s inference is correct given what it can see, and what it can see is wrong. This is exactly the stale-replica problem from the replication module, with one aggravating property: the replica cannot represent its own incompleteness. A database replica knows its log position and can tell you it is behind. A truncated context contains no marker saying "three steps were removed here". The model has no way to know it is missing something, and therefore no way to hedge.

The consequence is a family of bugs that all look like the agent being unreliable: it repeats a step it already completed, contradicts a decision it made earlier, asks the user for information already given, or abandons a constraint from the original instruction. None of these are model quality problems in the usual sense — they are consistency problems, and the fixes are the ordinary ones. Keep the durable step log authoritative, re-derive the important facts into the context on every turn rather than relying on them surviving, and make the constraints that must not be lost part of the system prompt rather than part of the history.

A tool returns {"error": "rate limited, retry after 30s"}. The orchestrator appends it to the context as a tool result. The model reads it as information and writes it into the report: the summary now contains a sentence about rate limiting where a figure should be. Or a proxy returns an HTML error page, and the model, asked to extract a value, extracts something from the error page.

The structural cause is that in this architecture the error channel and the data channel are the same channel. In an ordinary program an exception propagates out of band; here every result — success, failure, partial, malformed — arrives as text in the same slot. Nothing separates them unless the tool contract does.

So the fix belongs at the boundary, not in the prompt. Validate every tool result against a schema before it enters context. Make failure a distinct, typed shape the orchestrator handles rather than a string the model reads. Never pass raw upstream error bodies through — map them to a small controlled vocabulary that the orchestrator understands and the model is told how to treat. tool-output-untrusted in Security makes the same argument from the trust direction, and it is the same boundary: what a tool returns is untrusted input to be validated, not a value to be consumed.

The same discipline handles the case where a tool succeeds but returns nothing useful — an empty search, a null field. Empty is a legitimate result and must be distinguishable from an error, or the model will treat one as the other and either fabricate or give up.

Three more, each a familiar distributed problem wearing agent clothing.

Fan-out amplification. An orchestrator spawns five subagents, each of which calls three tools, each with three retries, and each subagent may itself spawn. The multiplier is a product nobody chose, and against a rate-limited provider it becomes [[retry-amplification]] with a large constant. The mitigations are the ordinary ones — [[retry-budgets]], [[backoff-and-jitter]], and a depth cap — plus one that is specific: a *global* budget per task, in tokens and tool calls, that terminates the workflow regardless of what any individual component believes. That budget is the only backstop against a failure that produces no errors and does not stop on its own.

Silent degradation. A fallback to a cheaper or smaller model on provider failure produces structurally valid output at lower quality. Latency improves, cost improves, error rate is zero, and the product gets worse. No operational metric catches this — it is only visible in evaluation, which is why fallbacks-caching-routing and evaluation belong to the same conversation. At minimum, record which model actually served each call, so the correlation is available after the fact.

Poison tasks. A task that always fails — malformed input, an impossible instruction, a permanently broken tool — is retried forever, consuming budget and worker capacity. This is [[poison-messages]] exactly, and the answer is unchanged: bound the attempts and move it to a [[dead-letter-queues]] destination for inspection. Agent systems get this wrong more often than ordinary consumers because the failure is not an exception — the task "completes" each time with an unsatisfactory result, and the loop tries again.

The unifying property of this lesson is that none of it shows up in the three signals most systems monitor: error rate, latency, saturation. So the instrumentation has to be built specifically, and it is a short list.

One correlation id per workflow, propagated into every model call, tool call, memory write and subagent, so a trace can be assembled at all — [[correlating-distributed-logs]]. Without it, every investigation starts by trying to work out which log lines belong together.

Recorded model outputs and tool results, retained long enough to investigate, because the run cannot be reproduced by re-running it.

Cost and token count per completed task, which is the single best leading indicator in this whole module: loops, amplification, context bloat and retry storms all raise it before anything else moves.

Action-sequence assertions: did this workflow call the irreversible tool more than once, call tools outside its permitted set, or exceed its depth? These are cheap, deterministic checks over the trace and they catch a surprising share of real bugs.

Outcome evaluation on a sample, because structural correctness does not imply a correct answer, and the failures in this lesson are exactly the ones that are structurally fine.