Agent + RAG
An agent whose knowledge lives outside the model: retrieval is either a tool the agent chooses to call or a step that always runs before the model sees the question.
Two ways to attach retrieval
Retrieval-augmented generation puts documents into the context at request time instead of relying on model weights. There are two architectural placements, and they behave very differently. Always-on retrieval runs a search before every model call: query → top-k chunks → prompt → answer. Retrieval as a tool exposes search_knowledge(query) alongside other tools and lets the agent decide whether, when, and how many times to search.
Always-on is a pipeline, not an agent: deterministic, one model call, easy to evaluate with a fixed golden set as in RAG Evaluation. Retrieval-as-a-tool is agentic: the model may reformulate the query, search three times, or skip retrieval when the question is chit-chat. It is more capable and strictly harder to test.
A common hybrid: always-on retrieval for the first turn (cheap, guarantees grounding), plus a search tool for follow-ups. That gives you a predictable baseline with an escape hatch.
Search vs fetch: two tools, not one
Give the agent two retrieval tools with different contracts. search_knowledge(query, k=5, filters) returns ranked snippets with ids and scores — small, lossy, fast. retrieve_documents(ids) returns full sections for the ids the agent decides matter. This mirrors how a person uses a search engine: skim results, then open two of them.
The split keeps context small. A search returning five 300-token snippets costs 1.5k tokens; pulling five full documents costs 20k. Letting the model choose which to expand is the cheapest context-engineering lever in the whole system, as Context Selection & Compression shows.
Under the hood, search_knowledge should be hybrid — BM25 for exact identifiers and product codes, dense vectors for paraphrase — followed by a reranker when latency allows. Which retrieval mode to use is a retrieval question (Dense, Sparse & Hybrid Retrieval); the architecture question is only where the call sits.
search_knowledge: query → top-k snippets withdoc_id,chunk_id, score, and metadata. Keep k ≤ 8.retrieve_documents: ids → full text, capped per call (for example 4 documents, 3k tokens each).- Both tools return untrusted content: anything in a retrieved document must be treated as data, never as instructions.
- Return citations (
doc_id, span) with every snippet so the final answer can cite, per Citations.
Choosing the placement
Choose always-on when nearly every request needs the knowledge base, when latency budgets are tight (one retrieval + one model call ≈ 1–2 s), and when you want offline evaluation to be trivial. This is the right shape for documentation Q&A, policy lookup, and most support deflection.
Choose retrieval-as-a-tool when questions are heterogeneous — some need documents, some need the CRM, some need nothing — or when a single search is often insufficient and the model benefits from reformulating. Multi-hop questions ("which customers on the enterprise plan are affected by the incident in last week's postmortem?") need two searches with the second depending on the first.
Do not let the agent decide when the cost of a wrong decision is high. If the agent skips retrieval and answers from weights, it will hallucinate confidently; if you cannot tolerate that, force retrieval and give the model a "no relevant documents found" path instead.
1const tools = {2 search_knowledge: {3 description: 'Search internal docs. Returns ranked snippets with doc ids. Use before answering factual questions.',4 schema: { query: 'string', k: 'number?', filters: '{ product?: string, updatedAfter?: string }?' },5 run: async ({ query, k = 5, filters }) => hybridSearch(query, { k, filters }),6 },7 retrieve_documents: {8 description: 'Fetch full sections for doc ids returned by search_knowledge. Max 4 ids.',9 schema: { ids: 'string[]' },10 run: async ({ ids }) => store.getMany(ids.slice(0, 4)),11 },12};13// Always-on variant: prepend retrieval, then a single model call.14async function answerAlwaysOn(question: string) {15 const hits = await hybridSearch(question, { k: 6 });16 if (hits.length === 0) return { text: 'No relevant documents found.', citations: [] };17 return llm({ system: GROUNDED_SYSTEM, context: hits, question });18}Properties and costs
Complexity is moderate: you now own an ingestion pipeline, an index, an embedding model version, and a freshness process, on top of the agent. Most of the engineering time in an agent+RAG system goes into the RAG half — chunking, metadata, and evaluation — not into the agent.
Latency adds one retrieval round trip (typically 50–300 ms for hybrid search plus 100–500 ms for a reranker) per search. Retrieval-as-a-tool can trigger several searches per request, each with its own model call in between, so p95 is dominated by how many searches the agent chooses to run. Cap it.
Reliability is higher than a bare agent on factual questions because answers are grounded, but it introduces a new failure class: retrieval that returns plausible-but-wrong chunks, which the model then confidently paraphrases. Debuggability is good if you log the retrieved ids and scores on every span; without that, you cannot distinguish "bad retrieval" from "bad generation".
- Complexity: 3 — the index and ingestion pipeline are a second system to operate.
- Latency: 2–3 — one extra hop per search; multiple searches per request in tool mode.
- Cost: 2–3 — retrieved context is the main token cost; keep k small and expand on demand.
- Reliability: 4 on in-corpus factual questions, provided retrieval quality is measured.
- Debuggability: 4 — log query, ids, scores, and which snippets were cited.
Key points
- Two placements: always-on retrieval (a pipeline, one model call) and retrieval as a tool (agentic, multi-search).
- Split retrieval into
search_knowledge(cheap snippets) andretrieve_documents(expand by id) to keep context small. - Retrieved content is untrusted data; it must never be treated as instructions.
- Always-on wins on latency, testability, and guaranteed grounding; tool mode wins on heterogeneous and multi-hop questions.
- Cap the number of searches per request — p95 latency and cost scale with it.
- Log retrieved ids and scores on every trace or you cannot separate retrieval failures from generation failures.
When to use — and when not to
- Answers depend on private, changing, or voluminous documents the model was not trained on.
- You need citations or auditability for every factual claim.
- Questions span several capabilities (documents, CRM, tickets) and the agent must pick between them.
- Multi-hop questions where the second query depends on the first result.
- The knowledge fits in the system prompt (under ~20k tokens and stable) — just include it.
- Every request needs the same documents — always-on retrieval or a cached context beats an agent deciding.
- You cannot invest in evaluating retrieval quality; ungrounded RAG is worse than an honest "I do not know".
- The corpus is untrusted and the agent has write-capable tools without an approval gate.
Failure modes
- Agent never calls the search tool because the tool description is vague, so answers come from weights.
- Retrieved chunks are topically similar but factually wrong for the question; the model paraphrases them confidently.
- Context bloat: the agent expands every hit to full documents and pushes the question out of attention.
- Stale index: documents changed but embeddings did not, producing answers that were true last quarter.
- Indirect prompt injection via a retrieved document that contains instructions.
- Search loops: the agent reformulates the same query six times because nothing matches.
Tradeoffs
Ratings assume retrieval quality is measured; unmeasured RAG is reliability 2.