Reading an Agent Run as a Trace
Model 2.2s, search 0.8s, model 1.6s, database tool 0.2s, model 1.4s — 6.2 seconds in a straight line. The question a waterfall answers is which of those steps are sequential because the data requires it, and which are sequential because that is the order the model happened to emit them.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The run as a waterfall
Instrument each step as a span — one per model call, one per tool call, nested under a run-level span — and an agent run becomes an ordinary trace you can read with ordinary skills (Reading the Waterfall). The value is immediate: the sequential structure that was implicit in the loop becomes a picture, and the picture makes the wasted time obvious in a way that a duration number never does.
The example below is a five-step run at 6.2 seconds. Everything is on the critical path, because everything is sequential — that is the default shape of an agent loop, and it is the shape worth questioning. Two of these steps are independent lookups: the search and the order-record fetch do not depend on each other's results. They are sequential only because the model emitted one tool call, received a result, and then emitted the other.
Recognising that turns a 6.2-second run into a 5.4-second run with no change to the model, the prompt or the tools — just concurrency where the data allows it (Parallel vs Sequential Tool Calls). And the same reading identifies the opposite case: steps that look parallelizable but are not, because the second query genuinely needs an id that only the first can supply.
Data dependency versus habit
The test for whether two steps can be concurrent is simple: does the input of the second contain any output of the first? If the order id comes from the user's original request rather than from the search results, the fetch never needed to wait. If the search query is constructed from a field in the order record, it genuinely did.
Agent frameworks serialize by default because the loop is naturally sequential — the model produces one message, you execute what it asked for, you send the result back. Getting parallelism requires either a model that emits multiple tool calls in a single message (and a framework that executes them concurrently), or your own code recognising a known pattern and pre-fetching. Both are worth doing, and neither happens by accident.
There is a useful middle path that is often overlooked: speculative prefetch for lookups the agent almost always makes. If 90% of runs on this task fetch the order record, fetch it before the first model call and put it in context. You pay for the 10% of wasted lookups and remove a full round trip from the other 90% (Every Optimization Buys Something and Sells Something — this buys latency and sells some wasted capacity). Whether that trade is good depends on the hit rate and the cost of the lookup, which you can measure.
1step 1 model.generate 2.2s → asks for search2step 2 tool.search 0.8s3step 3 model.generate 1.6s → asks for get_order(id)4step 4 tool.get_order 0.2s5step 5 model.generate 1.4s → final answer6 ─────7 6.2s8 9note: get_order used an id from the ORIGINAL user message.10It never depended on the search result. The 1.6s model call11in step 3 existed only to ask for something we already knew12we would need.1prefetch (parallel, before the first model call):2 tool.search 0.8s ┐ concurrent3 tool.get_order 0.2s ┘ 0.8s wall clock4 5step 1 model.generate 2.4s → both results already in context6step 2 model.generate 1.4s → final answer7 ─────8 4.6s (‑26%)9 10costs: get_order is fetched even in the ~10% of runs that11do not need it. Measured hit rate 91%, tool cost 0.2s —12the trade is clearly worth it here, and it is a trade.Nothing about the model changed. The 1.6-second middle generation disappeared because it existed only to request data that was predictable from the user's message, and the two tool calls overlapped because neither consumed the other's output. The remaining chain is a real dependency: you cannot answer before you have the data.
What to put on the spans
A useful agent trace carries more than durations. Each model span should record input and output token counts, the model identifier, TTFT separately from total, the finish reason, and whether the output was a tool call or a final answer (Inside One Model Call: Queue, First Token, Generation). Each tool span should record the tool name, the argument shape (not the values, which may be sensitive), success or failure, and retry count (Tool Errors, Retries and Timeouts).
Those attributes are what make aggregate analysis possible. With them you can ask "which tool contributes most p95 latency across all runs", "which task types have the highest step count", and "how much does context size grow per step on average" — questions that need per-span attributes and are unanswerable from durations alone (Trace, Span, Attribute, Status).
A specific caution: do not put full prompts and full tool results into span attributes by default. It is enormously tempting, since debugging an agent usually means reading what it actually said. But prompts contain user data and tool results contain business data, both of which then sit in a tracing backend with wide read access and long retention (What You Just Wrote Into a Log Half the Company Can Read). Sample full payloads for a small percentage of runs, redact known-sensitive fields, and keep the always-on attributes to identifiers, sizes and outcomes (Sampling Without Throwing Away the Evidence).
| Span | Attribute | Question it answers | Always on? |
|---|---|---|---|
| run | task type, outcome, step count | Which task types are slow, expensive or failing | Yes |
| model | model id, input tokens, output tokens | Where cost accumulates; whether context is growing (What One Agent Run Costs, and Which Term Dominates) | Yes |
| model | TTFT separate from total duration | Provider queueing versus prompt size versus generation length | Yes |
| model | finish reason (stop, length, tool_call) | Whether output caps are truncating answers | Yes |
| tool | tool name, success/failure, retry count | Which tool contributes the most latency and the most retries | Yes |
| tool | argument shape / schema version | Whether schema mismatches are driving retries (Tool Schemas) | Yes |
| model | full prompt text | What the model actually saw — essential for debugging | No — sample and redact |
| tool | full result payload | What the model actually received | No — sample and redact |
Key points
- Instrument each step as a span and the agent loop becomes an ordinary trace, readable with ordinary critical-path skills.
- The test for concurrency is whether the second step's input contains any of the first step's output — if not, the sequence was accidental.
- Agent frameworks serialize by default; parallelism requires either multi-tool-call messages or deliberate prefetching in your own code.
- Speculative prefetch of predictable lookups trades wasted calls on the minority of runs for a removed round trip on the majority.
- Carry token counts, TTFT, finish reason and tool outcomes as span attributes; sample full prompts and payloads rather than always recording them.
Agent Run Trace
Change an input and watch which number moves — and which one does not.
No individual model call got faster. The run got faster because there were fewer sequential steps — which is almost always where agent latency actually lives. Note the second-order effect: fewer model calls also means the accumulated context is sent fewer times, so cost falls alongside latency.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Model → framework: the model emits one tool call in one message, so the framework executes exactly one tool and returns the result.
- 2Framework → loop: the loop sends the result back for another decision, adding a full model round trip whose only purpose is to request the next lookup.
- 3Loop → serialization: independent lookups end up separated by model calls, so their durations add rather than overlap.
- 4Serialization → context: each additional round trip appends to the context, so every later model call has a higher TTFT (Inside One Model Call: Queue, First Token, Generation).
- 5Context → total: the run duration becomes the sum of every step plus the compounding prompt-processing cost, when the data-dependency graph allowed a much shorter critical path.
- • "Every step is on the critical path, so nothing can be parallelized" — everything is on the critical path *because* it is sequential; the question is whether the data required that ordering.
- • "The model calls are the problem" — a middle model call that exists only to request predictable data is removable, which is a structural fix rather than a model-speed fix.
- • "Our framework handles parallel tool calls" — verify it. Many execute concurrently only when the model emits multiple calls in one message, which many prompts never elicit.
- • "Prefetching wastes calls" — it wastes them on the runs that do not need the data, and measuring the hit rate turns this from an objection into an arithmetic problem.
- • "We should log the full prompt on every span" — that puts user data in a tracing backend with wide access; sample it instead (What You Just Wrote Into a Log Half the Company Can Read).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • One span per step nested under a run span, so the waterfall shape and the critical path are directly visible ([[critical-path]]).
- • Per-span token counts and TTFT, which turn the trace into a cost and latency attribution simultaneously.
- • Per-tool p95 latency aggregated across runs — the slowest tool is usually one tool, and it is usually fixable by ordinary means.
- • Data-dependency annotations, or at minimum a review of which tool arguments derive from earlier results versus from the original request.
- • Prefetch hit rate, if you prefetch: the percentage of runs that actually used the speculatively fetched data.
- • Identify independent steps by checking whether each tool's arguments derive from an earlier result or from the original request, and execute the independent ones concurrently.
- • Prompt for multiple tool calls in one message where the task allows it, and confirm the framework actually executes them concurrently.
- • Prefetch high-hit-rate lookups before the first model call, removing a full decide-and-request round trip from the majority of runs.
- • Fix the slowest tool as an ordinary dependency — it is usually one tool, and usually an unindexed query or an uncached lookup ([[slow-query-workflow]]).
- • Add span attributes for tokens, TTFT, finish reason and tool outcomes so aggregate questions become answerable.
- • The waterfall shows overlapping spans where you introduced concurrency, and the critical path is measurably shorter than the sum of durations.
- • Run p95 latency falls by approximately the duration of the removed or overlapped steps — a smaller gain means something else became the constraint ([[bottleneck-migration]]).
- • Prefetch hit rate is high enough that wasted lookups cost less than the removed round trip saves, measured rather than assumed.
- • Task success rate is unchanged: parallelizing or prefetching must not change what the model sees in a way that degrades answers ([[eval-metrics]]).
- • Concurrent tool calls raise peak load on tools and complicate partial-failure handling when one of several fails.
- • Prefetching wastes work on runs that do not need it, and adds load to the prefetched dependency proportional to total runs rather than to runs that use it.
- • Rich span attributes increase trace storage and can leak sensitive data if payloads are included without redaction.
- • Restructuring the loop for parallelism reduces the model's freedom to choose its own path, which is sometimes exactly the flexibility the system was built for ([[architecture-tradeoffs]]).
- • Assert on critical-path span count in the eval suite, so a prompt change that reintroduces a serial round trip fails a gate (Regression Gates and Online Evaluation).
- • Alert on per-tool p95 latency independently, since a tool degrading turns into agent latency with no agent change.
- • Track prefetch hit rate continuously — a drift downward turns a good trade into wasted capacity.
- • Keep sampling of full prompts at a fixed low rate with redaction, so debugging remains possible without the payload volume growing unbounded (Sampling Without Throwing Away the Evidence).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe waterfall timings, the prefetch hit rate and the resulting percentages are invented to show the reasoning. Real step durations depend on model, provider, tools and question.
- RUNTIME-SPECIFICWhether independent tool calls execute concurrently depends on both the model emitting them together and the framework executing them that way. Both must be verified in your stack.