Toolsparallel tool callsfan-outdependency graphresult orderingconcurrency limits

Parallel vs Sequential Tool Calls

Run independent tool calls concurrently and dependent ones in order — a dependency graph, a fan-out limit, and deterministic result ordering keep it fast and debuggable.

Interview question
Progress

Independence decides the schedule

Modern providers let the model emit several tool calls in a single response. Whether you may execute them at the same time is not the model's decision — it is a property of the calls. Two calls are independent if neither needs the other's result and they do not contend for the same resource. get_weather("Paris") and get_weather("Rome") are independent. find_customer("acme") and list_orders(customer_id=?) are not: the second argument comes from the first result, so the model cannot even emit it until the first returns.

So the two shapes fall out naturally. Parallel: the model emits N independent calls in one turn; the dispatcher runs them concurrently and returns N results together. Sequential: the model emits one call, sees the result, emits the next. Latency is max(t_i) for parallel and sum(t_i) + N × model_latency for sequential — with model latency often dominating, a 5-call sequence can be 10× slower than the same 5 calls fanned out.

The dispatcher should still decide, not assume. Even calls the model emitted together can conflict: two update_ticket calls on the same id, or two writes to the same file. Group by resource and serialize within a group.

Dependency ordering is topological sort

When an agent or planner produces a batch of steps with explicit dependencies — step 3 needs the output of steps 1 and 2 — the execution schedule is exactly a topological order of the dependency DAG, and the parallelism available at each level is the set of nodes whose predecessors are all complete. Kahn's algorithm gives you both: repeatedly run every node with in-degree zero, then decrement its dependents.

Cycle detection comes for free: if the queue empties while nodes remain, the plan is circular and should be rejected before any tool runs. This is the same machinery as build systems and workflow engines; a Workflow State Graph is this DAG made explicit and persistent.

Level-by-level execution of a dependency DAG with a fan-out limit.
1type Step = { id: string; deps: string[]; run: (inputs: Record<string, unknown>) => Promise<unknown> }
2 
3async function executePlan(steps: Step[], maxParallel = 4) {
4  const results: Record<string, unknown> = {}
5  const remaining = new Map(steps.map((s) => [s.id, new Set(s.deps)]))
6  while (remaining.size > 0) {
7    const ready = [...remaining].filter(([, deps]) => deps.size === 0).map(([id]) => id)
8    if (ready.length === 0) throw new Error('cyclic dependencies in plan')
9    for (let i = 0; i < ready.length; i += maxParallel) {
10      const batch = ready.slice(i, i + maxParallel)               // bounded fan-out
11      const settled = await Promise.allSettled(
12        batch.map((id) => steps.find((s) => s.id === id)!.run(results)),
13      )
14      batch.forEach((id, k) => {
15        const r = settled[k]
16        results[id] = r.status === 'fulfilled' ? r.value : { error: String(r.reason) }
17        remaining.delete(id)
18        for (const deps of remaining.values()) deps.delete(id)
19      })
20    }
21  }
22  return results
23}

Fan-out limits and back-pressure

Unbounded parallelism is a self-inflicted denial of service. If the model decides to check 200 URLs, Promise.all over 200 fetches will trip rate limits, exhaust connection pools, and — if each result is 3 KB — push 600 KB into the next context window. Cap concurrency with a semaphore (4–10 is typical for HTTP tools, 1 for anything holding a DB lock) and cap the total number of calls per turn.

Beyond the limit, either queue (slower, complete) or truncate and tell the model ("ran 20 of 200 requested lookups; narrow the query"). Telling the model is usually better — it forces a re-plan rather than silently degrading. Per-tool concurrency limits belong in the tool registry alongside the retry class from Idempotency, so a new tool cannot forget them.

Cost and token budgets apply to fan-out just as to steps (Budgets, Limits and Termination): 50 parallel searches are cheap in wall-clock time and expensive in tokens.

  • Semaphore per tool (HTTP: 4–10, DB writes: 1, LLM sub-calls: 2–4).
  • Hard cap on tool calls per turn (e.g. 25) with an explicit observation when hit.
  • Bound the size of each result before it enters context; summarize or paginate large ones (Context Selection & Compression).
  • Use allSettled, not all: one failed lookup should not discard nine successes.

Ordering results in context

Parallel calls complete in arbitrary order, but the model needs a stable mapping from call to result. Providers require every tool result to carry the tool_call_id it answers, and most require results to be appended in the same order the calls were emitted. Do that regardless — deterministic ordering makes traces reproducible and evals comparable, and models attend to position (Context Ordering & Lost in the Middle).

Include the failed calls too, as structured errors, in their original slot. A missing result makes the model think the call never happened and re-issue it. If results are large, order also affects what survives truncation: put the result the next reasoning step most needs closest to the end.

Fan-out with bounded concurrency and ordered fan-in
LLM emits 3 callsSemaphore (max 2)call #1call #2call #3 (waits)Order by call idLLM reads results 1,2,3
UserLLMAgentToolDataDecisionHumanGuardrail

Key points

  • Parallelize only independent calls; dependence is a property of the data flow, not the model's mood.
  • Sequential chains pay a model round trip per call — the dominant latency term.
  • Explicit dependencies form a DAG; execute in topological levels, reject cycles before running anything.
  • Bound fan-out with per-tool semaphores and a per-turn call cap; tell the model when the cap is hit.
  • Return results in emitted order with tool_call_id, including failures as structured errors.
  • Watch tokens, not just wall clock: parallel calls are fast and expensive.

When to use — and when not to

Use it when
  • Batch lookups over many independent keys (ids, cities, URLs).
  • Gathering from several sources before a synthesis step (search + DB + calendar).
  • Planner output with explicit step dependencies that a deterministic executor can schedule.
Avoid it when
  • Writes to the same entity — serialize them or merge into one call.
  • When the second call's arguments depend on the first result; forcing parallelism yields guessed arguments.
  • Tools that share a rate-limited quota with production traffic.
  • When results are large and the synthesis step only needs a few of them — retrieve lazily instead.

Failure modes

  • Model emits list_orders(customer_id="<from previous call>") as a literal string in a parallel batch.
  • 200-way fan-out hits the upstream rate limit; every call fails and the model retries all 200.
  • Results appended in completion order; the model attributes Rome's weather to Paris.
  • One rejection in Promise.all discards nine good results and the turn is wasted.
  • Two parallel update_ticket calls race; last writer wins and one field change is lost.
  • Cyclic plan from the model executes forever because nobody checked in-degrees.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

Parallelism cuts wall-clock latency sharply but adds scheduling logic and makes traces harder to read.

← IdempotencyTool Permissions and Least Privilege →