Case: An AI Assistant for Company Documentation
"Answer questions about company documentation" → Documents, Retrieval, Context, Model, Answer, Evaluation. The first version is retrieval plus one prompt, measured against a question set; the why ladder for "we need a multi-agent system" ends at a retrieval fix most of the time — and says when it would not.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
You are asked for an assistant that answers questions about internal documentation. What are the six capabilities, which one do people skip, and what would have to be true before "we need agents" is the right next move?
Support keeps answering the same questions from the internal handbook, and someone wants "an AI that knows our docs". The demos you have seen are agents with tools and memory and a planner, and it is hard to tell what a first version even is. You also cannot say, right now, how anyone would know whether the assistant's answers were good.
Build the agent. A planner that decides which tool to call, a retriever tool, a summariser, memory across turns — the architecture from the demos. It handles every case you can imagine, and the framework makes wiring it up an afternoon.
There is no question set, so there is no way to say whether the agent is better than a single prompt over the right paragraph — and it usually is not, because the hard part was finding the paragraph.
- There is no question set, so there is no way to say whether the agent is better than a single prompt over the right paragraph — and it usually is not, because the hard part was finding the paragraph.
- The planner's decisions are invisible. When an answer is wrong, the cause could be retrieval, the prompt, the model, the planner's tool choice or the memory, and nothing was built to tell them apart.
- The demo's capabilities are solutions to problems the handbook does not have. Multi-step planning solves tasks that need several actions; the handbook needs one: find the passage and answer from it.
- Evaluation is the capability everyone skips, so the assistant ships on the strength of five impressive answers, and the wrong ones are found by the support team it was meant to relieve.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Decompose into six capabilities and build them in order: Documents (what is in scope, how it is split), Retrieval (finding the passages a question needs), Context (assembling them into what the model sees), Model (the call), Answer (what the user gets, with its sources), Evaluation (a question set with known-good answers, scored). Evaluation is sixth in the list and first in importance, because it is how every other capability is judged (Decomposition by Capability).
- Build the smallest version that can be evaluated: a fixed set of documents, chunked; retrieval by embedding similarity; the top passages in a prompt that says to answer only from them and to cite; one model call. Then run the question set and look at the failures, one at a time, and attribute each to a capability (The Smallest Executable Thing).
- Ask "why is one deterministic workflow insufficient?" before adding any agentic structure. Most failures in a documentation assistant are retrieval failures — the right passage was not in the context — and no amount of planning fixes that. The why ladder for agents has to pass through the evaluation results (Problem Solving and Agentic Systems).
- Treat the evaluation set as a product requirement and grow it from real questions: the ones support answers, the ones the assistant got wrong. A version that scores better on it is an improvement; anything else is a demo (Experiment Design).
Six capabilities, evaluation first
The decomposition puts Evaluation last in the tree and first in the practice, because it is the observation every other leaf is tested against. Each leaf has its own test; Retrieval's is deliberately measured without the model, because a retrieval miss cannot be fixed downstream.
- ├Documents— what is in scope, and how it is split into passages
- └Scope and chunkingtestable Every handbook section is present as passages that stand alone; a procedure is not cut in half; out-of-scope documents are absent.
- ├Retrieval— finding the passages a question needs — where most failures live
- └Known source in the top resultstestable For each question in the set, the passage that answers it is among the retrieved passages; measured without the model.
- ├Context— what the model actually sees
- └Passages assembled with instructionstestable The prompt contains only retrieved passages and the rule to answer from them; a question with no relevant passage yields an empty context, not a guess.
- ├Model— the call — one of six, not the whole thing
- └Answer from contexttestable Given the right passage, the answer matches the known-good answer; given none, it says so.
- ├Answer— what the user receives
- └Cited, or declinedtestable Every answer names its passage; "not in the docs" appears for questions the handbook does not cover (Citations).
- ├Evaluation— how every other leaf is judged
- └Question set, scored per capabilitytestable A change to any leaf produces a before-and-after score, and a failure is attributed to one leaf.
Nothing in the tree plans, remembers or calls tools. Those are capabilities for a different task, and they arrive with their own evaluation.
"We need a multi-agent system"
The ladder as it was run, with the rung where the evaluation results answered the question. It ends with the case where agentic structure was justified — a task, not a question — because the device is not "agents are wrong"; it is "find out what you actually need".
“We need a multi-agent system for the documentation assistant.”
- ↓Why agents? The assistant gets things wrong and a planner could reason about which tool to use.
- ↓Which things does it get wrong? The evaluation shows retrieval misses on questions in the user's words, and answers that blend two teams' policies.
- ↓Would a planner fix a retrieval miss? No — it would call the same retriever with the same question.
- ↓Would a second agent fix the blended policies? No — the fix is knowing the asker's team and filtering passages by it, which is a metadata filter.
the claim was right when The task is multi-step and stateful — file the request, not explain it — needing tool calls, state across steps and confirmation. Then a single agent loop with a small tool set is the first version, and several agents are justified only when one context cannot hold the task or when parts of it must run with different permissions (Multi-Agent Systems Overview, Tool Permissions and Least Privilege).
The first version, as a slice
The slice is retrieval plus one prompt, run against the question set. What it proves is precise, and what it does not prove is the list of capabilities the demos led with. The learning map at /thinking/learn-map turns this list into the domains each one needs.
- DocumentsThe handbook, chunked into standalone passages, embedded once.
- RetrievalEmbed the question; return the top few passages by similarity, filtered by the asker's team.
- Context + ModelOne prompt: the passages, the rule to answer only from them and cite, the question; one call.
- AnswerThe answer with its passage named, or "not in the docs".
- EvaluationScore retrieval hit rate and answer correctness on the question set; record both.
How to do it
Most important first.
- Collect the question set first — the questions support actually answers, with the handbook passage that answers each. A few dozen is a start; what matters is that each has a known-good answer and a known source (Example-Driven Thinking).
- Documents: decide what is in scope and split it into passages that stand alone. Try chunking on the real handbook and read the chunks; a chunk that cuts a procedure in half is a retrieval failure waiting to happen (Ingestion: Parsing & Chunking in Agentic covers it).
- Retrieval: embed, search, return the top few. Measure it on its own — is the known source in the top results for each question? — before involving the model at all.
- Context and Model: one prompt that instructs answering only from the passages and citing them; one call. Answer: show the citation, and say "not in the docs" when the passages do not cover the question.
- Evaluation: score retrieval hit rate and answer correctness separately, on every change. Attribute failures to a capability before fixing anything (RAG Evaluation in Agentic is the depth).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- The question set, before any code. Support's recurring questions — expense limits, how to request leave, who approves a purchase — with the handbook section that answers each. Two things were learned before a model was called: several questions had no answer in the handbook, which is a documentation gap and not the assistant's job; and several answers depended on which team you were on, which means the assistant needs to know the asker's team or must ask.
- The first version, measured. Chunked handbook, embedding retrieval, one prompt, one call. Retrieval put the known source in the top results for most questions; the answers from those were mostly correct. The failures split cleanly: retrieval misses on questions phrased in words the handbook does not use, and a few answers that blended two teams' policies because both passages were retrieved. Neither failure is a planning failure.
- The why ladder for agents, run when someone proposed it. "We need a multi-agent system" → why? → the assistant gets things wrong → which things? → the retrieval misses and the blended policies → would a planner find a passage the retriever missed? → no; it would call the same retriever. Real requirement: find the right passage for questions in the user's words, and answer for the user's team. Simpler: rewrite the query before retrieval, add keyword search beside embeddings, and filter passages by the asker's team. The evaluation score moved; the agent would not have.
- The case where it would be justified, so the ladder is honest. Support later asked for the assistant to file the leave request, not just explain it — a multi-step task with tool calls, state across steps and a confirmation. That is a different capability with a different evaluation, and a single-agent loop with a small tool set is its first version; several agents are justified only when one context cannot hold the task, which the readings did not show (When Not to Use Multi-Agent and Single Agent draw the line).
How you know it worked
What now exists that did not before, and what question you can now ask.
- A question set with known-good answers and sources exists, and every version is scored against it before anyone sees a demo.
- Retrieval is measured separately from answering, and a wrong answer can be attributed to one of the six capabilities.
- The assistant says "not in the docs" for questions the handbook does not answer, and the documentation gaps it found have been handed to whoever owns the handbook.
- You can state what a planner or a second agent would have to do that the retriever cannot, and you can say whether the evaluation shows that need.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What would a known-good answer look like for this question, and where in the source does it come from?
- ?Is this failure retrieval, context, the model, or the answer format — and how would I tell?
- ?Why is one retrieval-and-answer step insufficient for this task, and what evidence shows it?
- ?What can the assistant not answer because the documentation does not say, and who owns that gap?
What can go wrong
- The question set is built from the assistant's own good answers, and the evaluation measures agreement with itself. Questions have to come from users and answers from the source.
- Retrieval is tuned until the score is high on the set, and the set never grows, so the assistant is excellent on last month's questions.
- Agentic structure is refused on principle after the ladder, including for the leave-request task that needs a tool loop. The ladder ends at "justified when" for a reason.
- Evaluation exists but is run by hand and skipped under deadline; the version that shipped is the one nobody scored (Regression Gates and Online Evaluation).
- Building the question set first delays the first impressive demo by days, and the demo is what gets the project its sponsor. The case bets that the first wrong answer in front of the sponsor costs more.
- A single retrieval-and-answer step cannot do multi-step tasks; the case deferred those until they were asked for, which meant saying no to the leave-request feature for a while.
- Saying "not in the docs" is honest and unimpressive; a model that answers from general knowledge looks more capable and is wrong in ways the evaluation set is designed to catch.
- "Agents are over-engineering." A tool loop was the correct first version for the leave-request task, and several agents may one day be justified. The case argues against agentic structure as an answer to retrieval failures, which is what the first proposal was.
- "Evaluation means an accuracy number." The number matters less than the attribution: which capability failed on which question. A score without attribution says "worse" and not "why".
- "The model is the assistant." The model is the fourth capability of six. Most of the assistant's quality was decided by documents, retrieval and the question set, none of which the model choice affected.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALDocuments, Retrieval, Context, Model, Answer, Evaluation — with evaluation built first — applies to any assistant grounded in a body of text; the retrieval details differ by corpus and the evaluation differs by what a good answer means.
- DOMAIN-SPECIFICFor a documentation assistant, a wrong answer is a support ticket; for one advising on contracts or medication, a wrong answer is a liability, and the evaluation set, the citation requirement and the "not in the docs" path are not optional extras but the product. The case is the low-stakes version and says so.
- CONTESTEDA strong opposing view from agentic practitioners: retrieval-then-answer is a ceiling, not a floor — real questions need the assistant to decide what to look up, look up several things, notice a contradiction between two passages and ask a clarifying question, and building that reasoning loop from the outset avoids rebuilding the pipeline when the first multi-step question arrives. On that view the agent with a retriever tool is the smallest honest version, and the single prompt is a demo that will be thrown away.
- ILLUSTRATIVEThe handbook, support's questions, the retrieval findings and the leave-request feature are invented for the shape of the argument; no hit rates or scores are being reported.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's "review the LLM's answer" page at /manifesto/review is the evaluation habit at the scale of one answer; the question set is the same habit at the scale of a product.