Concurrency in Agent Systems

Parallel Tool Calls: Lower Latency, and Four Bills

Running three tool calls at once instead of in sequence turns three round trips into one. It also triples the instantaneous load on whatever they hit, opens the possibility of conflicting writes, raises the spend for calls whose results get discarded, and makes the run nondeterministic. All four costs are real; the latency win usually still wins for reads.

▶ Run the lab

The question this answers

The question

What exactly am I buying and paying for when I let the agent run tool calls in parallel?

The work

A research turn issuing six independent read-only lookups — three searches and three document fetches — where each call takes roughly 700ms.

What is shared

The search provider's rate-limit budget, the run's spend budget, and the agent's context, into which all six results are appended in some order.

The invariant — what must stay true under every interleaving

Every dispatched call is one the run still needs, the context contains each result exactly once, and the run stays inside its rate and spend budgets.

Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.

WorkCan it overlap?Can it parallelise?What is shared?What ordering?What synchronization?Where is contention?What can deadlock?What can race?What is gained?What complexity?

The win, quantified

Six sequential calls at 700ms each is 4.2 seconds. Six concurrent calls is roughly 700ms plus the slowest tail — call it 900ms. That is a 3.3-second reduction on a turn where the model itself might take 2 seconds, so it is a genuine and user-visible improvement, and for read-only lookups there is essentially no argument against it.

The shape of the win is the fan-out shape from Fan-Out / Fan-In: One Request Becomes N: total time becomes the *maximum* of the calls rather than the sum, which means it is bounded below by the slowest one. That has a specific consequence — parallel fan-out converts a latency problem into a tail-latency problem, because you now wait for the worst of six rather than the average of six. Six calls with a p99 of 3 seconds gives a turn whose p99 is dominated by whichever call was unlucky. See Fan-Out: Waiting for the Slowest of Seven and Tail Latency: Why p50 Being Fine Does Not Help.

It also means adding more parallel calls has sharply diminishing returns and rising tail risk. Going from six to twelve concurrent lookups does not make the turn slower in the mean, but it roughly doubles the chance that at least one of them is a slow outlier, which is what the user actually feels.

Six 700ms lookups, sequential against parallel. Note where the tail lands.ILLUSTRATIVE
Sequential — sum of all six
call 1
call 2
call 3
call 4
call 5
call 6 — total 4.2s
Parallel — max of all six
calls 1-5 complete (700ms)
call 6 is the slow one — 900ms total
model resumes
Search provider, instantaneous rate
sequential: 1 concurrent request
parallel: 6 at once, then nothing
Spend on discarded work
sequential: stop early if call 2 answers it
parallel: all six paid for regardless
↑ parallel turn done↑ sequential turn done
runningreadywaitingblockedidle1 tick ≈ 350ms

The four bills

Load. The same total work arrives as a burst rather than a stream. One run at six-wide is nothing; forty concurrent runs at six-wide is 240 simultaneous requests, which is a fan-out amplification the downstream experiences as a spike — Parallelism Moves the Load Downstream. The provider's rate limit is per key, shared across every run, and the failure appears only under concurrency that single-run testing never produces.

Conflicting writes. Nothing above applies to reads. The moment two parallel calls mutate the same resource, you have the full lost-update problem with an LLM choosing the schedule — covered in Two Agents, One Document. The practical rule is simple and worth being absolute about: parallelise reads freely, serialize writes by default, and require an explicit declaration to do otherwise.

Spend. Sequential execution can exit early — if call 2 answers the question, calls 3 through 6 are never issued and never billed. Parallel execution commits to all of them up front. For cheap search calls that is negligible; for expensive tools or sub-agent invocations it is a direct multiplier on cost per turn, and it is the least-noticed of the four because it does not fail, it just bills. See What One Agent Run Costs, and Which Term Dominates and Token Budgets.

Nondeterminism. Results arrive in completion order, which varies run to run. If they are appended to context in arrival order, the model sees a different context each time and may produce a different answer to an identical question. That defeats caching, complicates evaluation, and makes bug reports irreproducible — Nondeterminism: Same Input, Different Output and Determinism: Same Input, Same Output?. The fix is canonicalization: append in dispatch order regardless of completion order, which costs a small buffer and removes the whole class.

DimensionSequentialParallelWhich wins
Latency (6 x 700ms)~4.2s~0.9sParallel, decisively
Tail latencyAverage of sixWorst of sixSequential — parallel converts this into a tail problem
Instantaneous downstream load1 concurrent6 concurrent per run, x runsSequential
Early exitStop as soon as the answer is foundAll six committed and billedSequential
Cost per turnPay only for what was neededPay for everything dispatchedSequential
Write safetyNaturally serializedRequires explicit conflict handlingSequential
DeterminismFixed order, reproducibleCompletion order varies unless canonicalizedSequential, unless you canonicalize
Reads with no early exitSlow for no benefitStrictly betterParallel
Parallel versus sequential, honestly.

The shape that gets it right

The practical policy is not "parallel" or "sequential" but a small set of rules that a runtime can enforce: parallelise read-only calls, bound the fan-out with a semaphore, canonicalize result ordering, serialize mutating calls unless declared independent, and give every call a timeout so one slow outlier cannot hold the turn.

The timeout point is the one most often skipped and it matters most here. In a sequential run, a hung call blocks one step. In a parallel fan-out, a hung call blocks the entire turn because the fan-in waits for everyone — this is the Promise.all & gather semantics question, and whether your gather aborts on first error or waits for all is a decision with very different failure behaviour. A per-call timeout plus a partial-result policy converts "the turn hangs" into "five of six results, and the model is told so".

The code below is the shape. Note that the bound, the timeout, the ordering and the read/write split are all explicit — none of them are defaults you can rely on.

1const MAX_CONCURRENT = 4 // bound the fan-out. Not optional.
2const PER_CALL_TIMEOUT_MS = 5_000 // one slow call must not hold the turn.
3
4async function executeToolCalls(calls: ToolCall[]) {
5 const reads = calls.filter(c => toolSchema[c.name].effect === 'read')
6 const writes = calls.filter(c => toolSchema[c.name].effect !== 'read')
7
8 // 1. READS: parallel, bounded, timed out, order canonicalized.
9 const sem = new Semaphore(MAX_CONCURRENT)
10 const readResults = await Promise.allSettled(
11 reads.map(async (call, i) => {
12 await sem.acquire()
13 try {
14 return { i, value: await withTimeout(invoke(call), PER_CALL_TIMEOUT_MS) }
15 } finally {
16 sem.release()
17 }
18 }),
19 )
20
21 // canonicalize: append in DISPATCH order, not completion order.
22 // Without this, identical inputs produce different contexts run to run.
23 const ordered = readResults
24 .map((r, i) => ({ i, r }))
25 .sort((a, b) => a.i - b.i)
26
27 // partial results are a first-class outcome, not an error:
28 // tell the model which calls failed rather than failing the turn.
29 for (const { i, r } of ordered) {
30 context.append(r.status === 'fulfilled'
31 ? formatResult(reads[i], r.value)
32 : formatFailure(reads[i], r.reason))
33 }
34
35 // 2. WRITES: sequential by default. Each carries an idempotency key so a
36 // retry is not a second effect. Stop on first failure - a later write
37 // may depend on an earlier one having succeeded.
38 for (const call of writes) {
39 const result = await invoke(call, { idempotencyKey: keyFor(call) })
40 context.append(formatResult(call, result))
41 if (isFailure(result)) break
42 }
43}
Parallel tool execution with the four costs actually handled.

Key points

  • Six sequential 700ms calls take 4.2s; six parallel take about 0.9s. For read-only lookups this is a clear win and there is little argument against it.
  • Parallel fan-out converts a mean-latency problem into a tail-latency problem: you now wait for the worst of six, not the average.
  • Four costs: burst load on the downstream, possible conflicting writes, spend on results you would have skipped, and nondeterministic result ordering.
  • Sequential execution can exit early and stop paying; parallel execution commits to every call up front.
  • The absolute rule: parallelise reads freely, serialize writes by default, and require an explicit declaration to overlap two writes.
  • Canonicalize result ordering by dispatch index rather than completion, which removes an entire class of nondeterminism for the price of a buffer.
  • A per-call timeout matters more here than anywhere: in a fan-in, one hung call holds the whole turn.

The loop, answered

Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.

How it works
  • The runtime partitions proposed calls into read-only and mutating using the declared effect class in each tool's schema.
  • Read-only calls are dispatched concurrently under a semaphore that bounds in-flight calls, with a per-call timeout.
  • Results are collected with settle-all semantics so a single failure yields a partial result set rather than aborting the turn.
  • Results are appended to context in dispatch order, not completion order, so identical inputs produce identical contexts.
  • Mutating calls run sequentially, each with an idempotency key, stopping on the first failure since later writes may depend on earlier ones.
Interleavings that matter
  • Six reads dispatched together; five return in 700ms and one in 900ms. The turn completes in 900ms and the model sees all six results in dispatch order.
  • One of the six hangs with no timeout: the fan-in waits indefinitely and the turn never completes, even though five results were available in under a second.
  • Forty concurrent runs each fan out six ways: 240 simultaneous requests against a 100/s limit, and every run receives 429s for calls that would have succeeded sequentially.
  • Results appended in completion order: run 1 sees [C, A, B], run 2 sees [A, C, B] for identical inputs. The model produces different answers and the evaluation suite reports flakiness.
  • Two parallel updateDocument calls: both read version 4, both write version 5, and one edit is silently lost — the parallel-writes case that the read/write split exists to prevent. See Two Agents, One Document.
What it guarantees — and does not
  • Parallel dispatch guarantees the turn takes at most the slowest call plus overhead, rather than the sum. It guarantees nothing about the slowest call.
  • A semaphore guarantees a bound on in-flight calls within one run. Across runs the bound must be shared or it is not a bound — A Mutex on Server A Does Nothing About Server B.
  • Settle-all semantics guarantee partial results are available; abort-on-first-error semantics guarantee the opposite. Know which your gather does — Promise.all & gather.
  • Canonical ordering guarantees a reproducible context for a given set of calls. It does not make the model deterministic.
  • Idempotency keys guarantee a retried write is not a second effect *at destinations that honour them*, which is not all of them.
Where contention appears
  • The provider rate limit is shared across every concurrent run, so per-run fan-out multiplies into a fleet-wide burst.
  • The semaphore itself is a contention point: with a low bound and a wide fan-out, calls queue and the latency win shrinks toward the sequential case.
  • Context assembly is serialized at the fan-in, and very large results make that assembly itself a cost.
  • Model-provider concurrency limits apply to parallel sub-agent calls, which contend with each other for the same account-level budget.
How it fails
  • A hung call with no timeout holding the entire turn, despite every other result being ready.
  • Rate-limit rejection under concurrent runs, invisible in single-run testing.
  • Lost updates from parallel writes to the same resource.
  • Nondeterministic context from completion-order appends, producing irreproducible answers and flaky evaluations.
  • Cost overrun from speculative calls whose results are discarded, which produces no error and shows up only on the bill.
  • Partial failure treated as total failure, aborting a turn where five of six results would have been enough — Partial Failure: When 3 of 5 Succeed.
When it helps
  • Read-only fan-out for research and retrieval, where the calls are independent, cheap, and there is no early-exit opportunity anyway.
  • Latency-sensitive interactive agents, where three seconds saved is directly perceptible.
  • Prefetching data the turn will certainly need, where speculation is not speculation.
When it hurts
  • When early exit was likely: paying for six calls to answer a question that call two would have answered.
  • With expensive tools or sub-agent invocations, where the cost multiplier is large and the latency saving modest.
  • On any mutating call, where the correct default is serialization and parallelism buys risk.
  • Under rate limits with many concurrent runs, where the fan-out converts throughput into rejections.
  • When turn latency is dominated by the model, so tool parallelism optimizes a minority of the time — Where an Agent Run Actually Spends Its Time.
How you would know
  • Turn latency p50 and p99 before and after — the p99 is where fan-out's tail cost shows up.
  • Concurrent in-flight tool calls, per run and across all runs, which is what the downstream actually sees.
  • Cost per turn split into used and discarded results, which quantifies the speculation bill.
  • Rate-limit rejections attributed to agent fan-out specifically.
  • Reproducibility rate: run identical inputs twice and compare contexts. Divergence means ordering is not canonicalized.
  • Timeout rate per tool, which tells you whether the fan-in is being held by a specific slow dependency.
Complexity it introduces
  • Effect classes in every tool schema, kept accurate as tools evolve — a mislabelled write is a silent correctness hole.
  • Two concurrency bounds (per run and fleet-wide), timeouts, and a partial-result policy the model must be told about.
  • Canonical ordering requires buffering results until all have settled, which increases peak memory per turn.
  • Evaluation and debugging must account for parallel execution; a trace of a parallel turn is harder to read than a sequential one — Trace Inspection: Debugging from a Trace.
Simpler alternatives
  • Sequential execution with early exit, which is simpler, cheaper, deterministic, and often fast enough — Parallel vs Sequential Tool Calls.
  • A batch tool that accepts multiple inputs in one call, giving the latency win with one request and no fan-out — Batch APIs and Partial Failure.
  • Caching read results across turns, which removes calls rather than overlapping them.
  • Speculative execution with cancellation: dispatch in parallel but cancel the rest as soon as one answers, recovering the early-exit saving — Cancelling an Agent Run: What Actually Stops.
  • Reducing model latency, which is usually the larger term and rarely the one that gets attention.

Agent parallel tool calls

Which of the agent’s tools can actually run at once?
Four tool calls in one turn. Mark the ones you believe are independent, then look at what the plan costs.
search_docs(query)
800 ms · reads [docs-index] · writes [—]
fetch_ticket(id)
600 ms · reads [ticket-db] · writes [—]
update_ticket(id, fields)
400 ms · reads [ticket-db] · writes [ticket-db] · needs fetch_ticket
post_comment(id, text)
300 ms · reads [—] · writes [ticket-db]
Sequential: 4 calls back to back2,100 ms
Your plan: 0 in parallel (0 ms), then 4 sequential2,100 ms
wall time
2100 ms
speedup
1.00×
peak downstream calls
1 at once
cost multiplier
1.00×
Nothing marked yet. Two of these four can genuinely run at once. One is blocked by a data dependency and one by a write conflict — and the difference matters, because the first is a correctness bug you can see in the transcript and the second is one you cannot.
0 marked parallelILLUSTRATIVE

What people believe, and what is true

Claim

Parallel tool calls are strictly better for latency.

Reality

They improve mean latency and worsen tail latency, because the turn now waits for the slowest of N rather than the average. Without per-call timeouts, one hung call holds everything.

Claim

Parallelism does not change cost, just timing.

Reality

Sequential execution can stop early. Parallel commits to every call, so the bill is the full fan-out whether or not the results were needed.

Claim

The results all arrive, so the order does not matter.

Reality

The model conditions on context order. Appending in completion order makes identical inputs produce different contexts and different answers.

Apply it