Hierarchical Pattern
Manager → team leads → workers scales a supervisor to many parallel sub-tasks, but each level compresses information and multiplies latency and cost.
Why add levels
A single supervisor with 30 specialists must hold 30 tool descriptions and merge 30 results; selection accuracy and context both suffer. A hierarchy groups specialists under leads: the manager picks among 4 leads, each lead among ~7 workers. This is the same reason organisations and B-trees have fan-out limits — a node can only manage so many children well.
The canonical use is large, decomposable work: "migrate 40 services to the new auth library" becomes a manager assigning services to backend/frontend/infra leads, each spawning one worker per service in parallel.
The compression tax
Information travels up as summaries. A worker that discovers "service B has a hard-coded token in a test fixture" writes it in a 400-token report; the lead condenses 6 reports into 300 tokens; the manager reads "backend: 6/6 migrated, minor issues". The security finding is gone. Design the upward schema to carry structured flags (blocking_issues: []) that cannot be summarised away.
Latency is the depth times the per-level round trip plus the slowest leaf. Two levels of 5 s coordination plus a 60 s worker is 70 s minimum. Cost is the total number of agent runs — for 40 workers and 4 leads and 1 manager, 45 agent loops, each with its own prompt overhead.
- Keep depth at 2; three levels is rarely justified and nearly impossible to debug from a trace.
- Fan-out per node of 5–10; beyond that, selection and merging degrade.
- Leaf workers should be the cheapest model that passes the worker eval; leads and manager can be stronger.
Deterministic alternative
Ask whether the hierarchy is really doing judgment at each level or just fan-out. If the manager's "plan" is "one worker per service", that is a for loop plus a queue, not an agent. Replace the coordination layers with code and keep LLMs only at the leaves; you get the same parallelism with zero compression loss and a trivially testable orchestrator (When Not to Use Multi-Agent).
1import asyncio2 3async def migrate_all(services: list[str]) -> list[dict]:4 sem = asyncio.Semaphore(8) # bound concurrency and cost5 async def one(svc: str) -> dict:6 async with sem:7 return await worker_agent.run(f"Migrate {svc} to auth-v2", budget_tokens=30_000)8 reports = await asyncio.gather(*(one(s) for s in services))9 blocking = [r for r in reports if r["blocking_issues"]] # structured, never summarised10 return blocking or reportsKey points
- Hierarchy fixes fan-out limits of a single supervisor.
- Each level compresses; carry critical findings as structured fields.
- Latency = depth × coordination + slowest leaf; cost = total agent runs.
- Depth 2, fan-out 5–10 is the practical envelope.
- If intermediate levels only fan out, replace them with code.
When to use — and when not to
- Dozens of parallel, similar sub-tasks with a few genuinely different groups.
- Sub-task groups need different leads with different domain prompts or tools.
- Batch workloads where a minute of latency is acceptable.
- Fewer than ~10 sub-tasks — a flat supervisor suffices.
- Intermediate levels make no decisions — use a queue and a loop.
- Interactive use cases.
Failure modes
- Critical findings summarised away on the way up.
- Cost explosion from unbounded worker spawning (
cost-explosion-after-launch). - One slow leaf stalls the whole tree without per-worker timeouts.
- Leads duplicate or contradict each other's decisions with no shared state.