The question this answers
What exactly am I buying and paying for when I let the agent run tool calls in parallel?
A research turn issuing six independent read-only lookups — three searches and three document fetches — where each call takes roughly 700ms.
The search provider's rate-limit budget, the run's spend budget, and the agent's context, into which all six results are appended in some order.
Every dispatched call is one the run still needs, the context contains each result exactly once, and the run stays inside its rate and spend budgets.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The win, quantified
Six sequential calls at 700ms each is 4.2 seconds. Six concurrent calls is roughly 700ms plus the slowest tail — call it 900ms. That is a 3.3-second reduction on a turn where the model itself might take 2 seconds, so it is a genuine and user-visible improvement, and for read-only lookups there is essentially no argument against it.
The shape of the win is the fan-out shape from Fan-Out / Fan-In: One Request Becomes N: total time becomes the *maximum* of the calls rather than the sum, which means it is bounded below by the slowest one. That has a specific consequence — parallel fan-out converts a latency problem into a tail-latency problem, because you now wait for the worst of six rather than the average of six. Six calls with a p99 of 3 seconds gives a turn whose p99 is dominated by whichever call was unlucky. See Fan-Out: Waiting for the Slowest of Seven and Tail Latency: Why p50 Being Fine Does Not Help.
It also means adding more parallel calls has sharply diminishing returns and rising tail risk. Going from six to twelve concurrent lookups does not make the turn slower in the mean, but it roughly doubles the chance that at least one of them is a slow outlier, which is what the user actually feels.
The four bills
Load. The same total work arrives as a burst rather than a stream. One run at six-wide is nothing; forty concurrent runs at six-wide is 240 simultaneous requests, which is a fan-out amplification the downstream experiences as a spike — Parallelism Moves the Load Downstream. The provider's rate limit is per key, shared across every run, and the failure appears only under concurrency that single-run testing never produces.
Conflicting writes. Nothing above applies to reads. The moment two parallel calls mutate the same resource, you have the full lost-update problem with an LLM choosing the schedule — covered in Two Agents, One Document. The practical rule is simple and worth being absolute about: parallelise reads freely, serialize writes by default, and require an explicit declaration to do otherwise.
Spend. Sequential execution can exit early — if call 2 answers the question, calls 3 through 6 are never issued and never billed. Parallel execution commits to all of them up front. For cheap search calls that is negligible; for expensive tools or sub-agent invocations it is a direct multiplier on cost per turn, and it is the least-noticed of the four because it does not fail, it just bills. See What One Agent Run Costs, and Which Term Dominates and Token Budgets.
Nondeterminism. Results arrive in completion order, which varies run to run. If they are appended to context in arrival order, the model sees a different context each time and may produce a different answer to an identical question. That defeats caching, complicates evaluation, and makes bug reports irreproducible — Nondeterminism: Same Input, Different Output and Determinism: Same Input, Same Output?. The fix is canonicalization: append in dispatch order regardless of completion order, which costs a small buffer and removes the whole class.
| Dimension | Sequential | Parallel | Which wins |
|---|---|---|---|
| Latency (6 x 700ms) | ~4.2s | ~0.9s | Parallel, decisively |
| Tail latency | Average of six | Worst of six | Sequential — parallel converts this into a tail problem |
| Instantaneous downstream load | 1 concurrent | 6 concurrent per run, x runs | Sequential |
| Early exit | Stop as soon as the answer is found | All six committed and billed | Sequential |
| Cost per turn | Pay only for what was needed | Pay for everything dispatched | Sequential |
| Write safety | Naturally serialized | Requires explicit conflict handling | Sequential |
| Determinism | Fixed order, reproducible | Completion order varies unless canonicalized | Sequential, unless you canonicalize |
| Reads with no early exit | Slow for no benefit | Strictly better | Parallel |
The shape that gets it right
The practical policy is not "parallel" or "sequential" but a small set of rules that a runtime can enforce: parallelise read-only calls, bound the fan-out with a semaphore, canonicalize result ordering, serialize mutating calls unless declared independent, and give every call a timeout so one slow outlier cannot hold the turn.
The timeout point is the one most often skipped and it matters most here. In a sequential run, a hung call blocks one step. In a parallel fan-out, a hung call blocks the entire turn because the fan-in waits for everyone — this is the Promise.all & gather semantics question, and whether your gather aborts on first error or waits for all is a decision with very different failure behaviour. A per-call timeout plus a partial-result policy converts "the turn hangs" into "five of six results, and the model is told so".
The code below is the shape. Note that the bound, the timeout, the ordering and the read/write split are all explicit — none of them are defaults you can rely on.
1const MAX_CONCURRENT = 4 // bound the fan-out. Not optional.2const PER_CALL_TIMEOUT_MS = 5_000 // one slow call must not hold the turn.3 4async function executeToolCalls(calls: ToolCall[]) {5 const reads = calls.filter(c => toolSchema[c.name].effect === 'read')6 const writes = calls.filter(c => toolSchema[c.name].effect !== 'read')7 8 // 1. READS: parallel, bounded, timed out, order canonicalized.9 const sem = new Semaphore(MAX_CONCURRENT)10 const readResults = await Promise.allSettled(11 reads.map(async (call, i) => {12 await sem.acquire()13 try {14 return { i, value: await withTimeout(invoke(call), PER_CALL_TIMEOUT_MS) }15 } finally {16 sem.release()17 }18 }),19 )20 21 // canonicalize: append in DISPATCH order, not completion order.22 // Without this, identical inputs produce different contexts run to run.23 const ordered = readResults24 .map((r, i) => ({ i, r }))25 .sort((a, b) => a.i - b.i)26 27 // partial results are a first-class outcome, not an error:28 // tell the model which calls failed rather than failing the turn.29 for (const { i, r } of ordered) {30 context.append(r.status === 'fulfilled'31 ? formatResult(reads[i], r.value)32 : formatFailure(reads[i], r.reason))33 }34 35 // 2. WRITES: sequential by default. Each carries an idempotency key so a36 // retry is not a second effect. Stop on first failure - a later write37 // may depend on an earlier one having succeeded.38 for (const call of writes) {39 const result = await invoke(call, { idempotencyKey: keyFor(call) })40 context.append(formatResult(call, result))41 if (isFailure(result)) break42 }43}Key points
- Six sequential 700ms calls take 4.2s; six parallel take about 0.9s. For read-only lookups this is a clear win and there is little argument against it.
- Parallel fan-out converts a mean-latency problem into a tail-latency problem: you now wait for the worst of six, not the average.
- Four costs: burst load on the downstream, possible conflicting writes, spend on results you would have skipped, and nondeterministic result ordering.
- Sequential execution can exit early and stop paying; parallel execution commits to every call up front.
- The absolute rule: parallelise reads freely, serialize writes by default, and require an explicit declaration to overlap two writes.
- Canonicalize result ordering by dispatch index rather than completion, which removes an entire class of nondeterminism for the price of a buffer.
- A per-call timeout matters more here than anywhere: in a fan-in, one hung call holds the whole turn.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The runtime partitions proposed calls into read-only and mutating using the declared effect class in each tool's schema.
- • Read-only calls are dispatched concurrently under a semaphore that bounds in-flight calls, with a per-call timeout.
- • Results are collected with settle-all semantics so a single failure yields a partial result set rather than aborting the turn.
- • Results are appended to context in dispatch order, not completion order, so identical inputs produce identical contexts.
- • Mutating calls run sequentially, each with an idempotency key, stopping on the first failure since later writes may depend on earlier ones.
- • Six reads dispatched together; five return in 700ms and one in 900ms. The turn completes in 900ms and the model sees all six results in dispatch order.
- • One of the six hangs with no timeout: the fan-in waits indefinitely and the turn never completes, even though five results were available in under a second.
- • Forty concurrent runs each fan out six ways: 240 simultaneous requests against a 100/s limit, and every run receives 429s for calls that would have succeeded sequentially.
- • Results appended in completion order: run 1 sees [C, A, B], run 2 sees [A, C, B] for identical inputs. The model produces different answers and the evaluation suite reports flakiness.
- • Two parallel
updateDocumentcalls: both read version 4, both write version 5, and one edit is silently lost — the parallel-writes case that the read/write split exists to prevent. See Two Agents, One Document.
- • Parallel dispatch guarantees the turn takes at most the slowest call plus overhead, rather than the sum. It guarantees nothing about the slowest call.
- • A semaphore guarantees a bound on in-flight calls within one run. Across runs the bound must be shared or it is not a bound — A Mutex on Server A Does Nothing About Server B.
- • Settle-all semantics guarantee partial results are available; abort-on-first-error semantics guarantee the opposite. Know which your gather does — Promise.all & gather.
- • Canonical ordering guarantees a reproducible context for a given set of calls. It does not make the model deterministic.
- • Idempotency keys guarantee a retried write is not a second effect *at destinations that honour them*, which is not all of them.
- • The provider rate limit is shared across every concurrent run, so per-run fan-out multiplies into a fleet-wide burst.
- • The semaphore itself is a contention point: with a low bound and a wide fan-out, calls queue and the latency win shrinks toward the sequential case.
- • Context assembly is serialized at the fan-in, and very large results make that assembly itself a cost.
- • Model-provider concurrency limits apply to parallel sub-agent calls, which contend with each other for the same account-level budget.
- • A hung call with no timeout holding the entire turn, despite every other result being ready.
- • Rate-limit rejection under concurrent runs, invisible in single-run testing.
- • Lost updates from parallel writes to the same resource.
- • Nondeterministic context from completion-order appends, producing irreproducible answers and flaky evaluations.
- • Cost overrun from speculative calls whose results are discarded, which produces no error and shows up only on the bill.
- • Partial failure treated as total failure, aborting a turn where five of six results would have been enough — Partial Failure: When 3 of 5 Succeed.
- • Read-only fan-out for research and retrieval, where the calls are independent, cheap, and there is no early-exit opportunity anyway.
- • Latency-sensitive interactive agents, where three seconds saved is directly perceptible.
- • Prefetching data the turn will certainly need, where speculation is not speculation.
- • When early exit was likely: paying for six calls to answer a question that call two would have answered.
- • With expensive tools or sub-agent invocations, where the cost multiplier is large and the latency saving modest.
- • On any mutating call, where the correct default is serialization and parallelism buys risk.
- • Under rate limits with many concurrent runs, where the fan-out converts throughput into rejections.
- • When turn latency is dominated by the model, so tool parallelism optimizes a minority of the time — Where an Agent Run Actually Spends Its Time.
- • Turn latency p50 and p99 before and after — the p99 is where fan-out's tail cost shows up.
- • Concurrent in-flight tool calls, per run and across all runs, which is what the downstream actually sees.
- • Cost per turn split into used and discarded results, which quantifies the speculation bill.
- • Rate-limit rejections attributed to agent fan-out specifically.
- • Reproducibility rate: run identical inputs twice and compare contexts. Divergence means ordering is not canonicalized.
- • Timeout rate per tool, which tells you whether the fan-in is being held by a specific slow dependency.
- • Effect classes in every tool schema, kept accurate as tools evolve — a mislabelled write is a silent correctness hole.
- • Two concurrency bounds (per run and fleet-wide), timeouts, and a partial-result policy the model must be told about.
- • Canonical ordering requires buffering results until all have settled, which increases peak memory per turn.
- • Evaluation and debugging must account for parallel execution; a trace of a parallel turn is harder to read than a sequential one — Trace Inspection: Debugging from a Trace.
- • Sequential execution with early exit, which is simpler, cheaper, deterministic, and often fast enough — Parallel vs Sequential Tool Calls.
- • A batch tool that accepts multiple inputs in one call, giving the latency win with one request and no fan-out — Batch APIs and Partial Failure.
- • Caching read results across turns, which removes calls rather than overlapping them.
- • Speculative execution with cancellation: dispatch in parallel but cancel the rest as soon as one answers, recovering the early-exit saving — Cancelling an Agent Run: What Actually Stops.
- • Reducing model latency, which is usually the larger term and rarely the one that gets attention.
Agent parallel tool calls
What people believe, and what is true
Parallel tool calls are strictly better for latency.
They improve mean latency and worsen tail latency, because the turn now waits for the slowest of N rather than the average. Without per-call timeouts, one hung call holds everything.
Parallelism does not change cost, just timing.
Sequential execution can stop early. Parallel commits to every call, so the bill is the full fan-out whether or not the results were needed.
The results all arrive, so the order does not matter.
The model conditions on context order. Appending in completion order makes identical inputs produce different contexts and different answers.