Architecture for a support bot
“Design the architecture for a customer support assistant that answers product questions and can process refunds. Which architecture do you choose and why?”
What this tests
- Applying the ladder to a realistic mixed task
- Separating read-only questions from risky actions
- Placement of retrieval, tools, and approval gates
- Thinking about evaluation and failure cost from the start
Answers by level
Read the beginner answer first and notice what is missing.
Two very different jobs are bundled here: answering questions (read-only, tolerant of mistakes) and moving money (irreversible, low tolerance). I split them at the front with a Router Architecture: a cheap classifier decides question, refund request, or escalate to human. Questions go to an Agent + RAG path: hybrid retrieval over product docs, reranking, grounded answer with citations. Refund requests go to a deterministic workflow: look up the order with a typed tool, check eligibility in code (not in the prompt), and gate the refund behind either hard rules or a human approval above a threshold. See Approval Gates and Risk Classes.
The refund tool itself should have a strict schema, argument validation against the order record, an idempotency key so retries cannot double-refund, and least-privilege scope (only orders belonging to the authenticated customer). The model never decides eligibility; it collects information and calls a tool that enforces policy.
Evaluation from day one: a golden set of questions with expected grounded answers, a set of refund scenarios with expected decisions, and tracing on every conversation so I can inspect misroutes.
Green flags · Red flags
- Separates read-only Q&A from irreversible actions
- Routes first with a cheap classifier and a confidence threshold
- Enforces refund eligibility in code, not in the prompt
- Adds idempotency, validation, least privilege to the refund tool
- Mentions approval gates for high-value refunds
- Plans evals and tracing from the start
- One agent with both docs search and a refund tool, policy in the system prompt
- No approval gate or threshold for money movement
- No thought about injection through customer messages
- No evaluation strategy