advancedAgents
A model returns a plan: a list of tool calls with arguments and dependencies. What does treating that as a program buy you?
Whether the candidate can transfer compiler discipline to a new setting rather than reaching for prompt engineering. The discriminator is whether they put the checks before execution rather than in a retry loop.
What a strong answer covers
- It gives you a place to put every check, before anything executes. The plan has a syntax, so you parse it rather than pattern-matching on strings, and a parse failure is a diagnostic you can feed back rather than a crash. It has a type discipline: each tool has a signature, so argument arity, types and enum membership are checkable statically, and a step that consumes the output of an earlier step can be checked for whether that output type is what this step accepts.
- It has a shape: the dependency structure is a graph, so you can reject cycles, find what is parallelizable, and refuse a plan whose step depends on something never produced — which is exactly name resolution over a symbol table of step outputs.
- And it has a policy layer that only exists because of all the above. Once you know statically which tools a plan calls and with what, you can check permissions before execution instead of at each call, refuse a plan that writes outside a sandbox, bound its cost, and require confirmation for the specific steps that need it. That is the difference between a validated plan and a hopeful one.
- The design consequence is to make the plan language small and deliberately non-Turing-complete — no arbitrary loops, no dynamic tool construction, no string-built arguments — because everything you can check statically is something the model cannot get wrong at run time. The place this is normally lost is a "run this code" tool, which reopens the whole surface.
✓ Green flags
- Maps the stages explicitly: parse, resolve, type-check, validate, then execute.
- Treats tool signatures as a type environment and checks against them.
- Names the dependency graph and cycle detection.
- Puts permission checks before execution, not inside the tool.
- Argues for restricting the plan language and knows what a code-execution tool costs.
✗ Red flags
- "Just validate the JSON schema." Schema validation checks shape and stops before cross-step dependencies, tool signatures and permissions.
- "Retry with the error message until it works." Retries are useful and are not validation — an invalid plan that happens to parse still executes.
- "The model is good enough now that this is overhead." The check costs microseconds and the failure costs a deleted bucket.
- "Make the plan language Turing-complete so it can express anything." Everything it can express is something you must then check at run time.
Follow-up
Step 3 uses the output of step 1 as a file path. What can you check statically, and what has to be checked at execution time?
Implementation challenge
What to ask them to write or trace on a whiteboard.
Define the validator for a five-step plan: list every check, and mark each as syntactic, type-level, dependency-level or policy-level.