Fundamentalsagent-loopstatetoolsterminationtokens

The Agent Loop

Every agent is the same loop — observe, reason, choose an action, run a tool, fold the result into state, check for completion — and every production problem lives in one of those boxes.

▶ InteractiveInterview question
Progress

The loop

Strip any agent framework down and you find one control structure. A goal enters. The agent observes its current state (conversation so far, tool results, retrieved context). The model reasons about what to do and chooses an action: call a tool with arguments, ask the user, or finish. The runtime executes the tool and receives a result. The result is appended to state. An evaluation decides whether the goal is met; if not, loop.

That is the whole thing. Frameworks add planning, memory, routing and guards, but they are decorations on these nine boxes. When an agent misbehaves, ask which box failed: bad observation (context missing), bad reasoning (prompt), bad action choice (tool schema), tool failure (retries), bad state update (context overflow), or bad evaluation (never terminates).

The agent loop
noyesGoalObserve stateReason / planChoose actionUse toolReceive resultUpdate stateGoal complete?Finish
UserLLMAgentToolDataDecisionHumanGuardrail

State, context and tools

State is everything the runtime knows: the message history, tool results, scratch variables, step count, spent tokens. Context is the subset of state that is actually placed in the model call this iteration. They diverge fast: after 30 tool calls the state may be 200k tokens while the context window you can afford is 30k. Deciding what goes in is Context Engineering.

Tools are typed functions the model can request: a name, a JSON schema for arguments, and a runtime handler. The model never executes anything; it emits a structured request and your code decides whether and how to run it. That boundary is where validation, permissions and approvals live — see Tool Calling Basics and Tool Permissions and Least Privilege.

A subtle point: the model chooses actions, but the runtime owns the loop. The while statement, the step counter, the budget check and the tool dispatch are ordinary code you control and can test deterministically.

  • State grows monotonically unless you compress or truncate it; context must fit a fixed window.
  • Tool results are untrusted input — a web page or email returned by a tool can carry instructions. See Indirect Prompt Injection.
  • The action set should be small: 5–15 well-described tools beat 60 vague ones for selection accuracy.

Termination

A loop with only "the model says it is done" as its exit is not finished engineering. Models under-terminate (keep "verifying"), over-terminate (declare success after a failed call), and occasionally oscillate between two actions. You need several independent stop conditions and the runtime must enforce them regardless of what the model wants.

  • Goal signal: the model returns a final answer instead of a tool call. Necessary, never sufficient.
  • Step limit: max_steps of 10–30 for most tasks; hitting it is a logged failure, not a silent success.
  • Token / cost budget: cumulative input + output tokens per run, converted to currency and capped.
  • Wall-clock timeout: for user-facing runs, seconds not minutes.
  • Repetition detector: identical tool call + arguments twice in a row means the agent is stuck; break and escalate.
  • Unrecoverable error: a permission denial or validation failure the model cannot fix by retrying.

Accounting per iteration

Each iteration is one model call whose input is the entire context so far. That makes cost roughly quadratic in the number of steps when context grows linearly: step *k* re-sends everything from steps 1..k-1. A 20-step run with 2k tokens added per step sends about 400k input tokens in total, not 40k.

Latency adds similarly: model time (proportional to output tokens plus prefill of the context) plus tool time per step, sequentially. Ten steps at 3 s each is a 30 s answer. Users tolerate that for a background job, not for a chat reply. Track tokens, latency and cost per step in a trace so you can see where a run got expensive — see Tracing Agents and Token Budgets.

  • Per step record: input tokens, output tokens, model latency, tool latency, tool name, whether the call was cached.
  • Prompt caching of the stable prefix (system prompt + tool schemas) cuts the quadratic term substantially.
  • Parallel tool calls in one step reduce latency but not tokens — see Parallel vs Sequential Tool Calls.

A minimal loop in Python

This skeleton is framework-free and shows every box of the loop explicitly. The llm function is a stand-in for any provider client that returns either a tool request or a final text; TOOLS maps names to handlers. Everything the model controls is in decision; everything you control is the rest.

A minimal agent loop with tool dispatch, step limit, token budget and stop conditions.
1from dataclasses import dataclass, field
2 
3@dataclass
4class State:
5    messages: list = field(default_factory=list)
6    steps: int = 0
7    tokens: int = 0
8 
9TOOLS = {
10    "search_invoices": lambda q: db.search(q),        # returns list[dict]
11    "get_customer":    lambda id: crm.get(id),
12}
13 
14def run(goal: str, max_steps: int = 15, token_budget: int = 200_000) -> str:
15    s = State(messages=[{"role": "user", "content": goal}])
16    last_call = None
17    while True:
18        if s.steps >= max_steps or s.tokens >= token_budget:
19            raise RuntimeError(f"stopped: steps={s.steps} tokens={s.tokens}")
20 
21        decision = llm(s.messages, tools=TOOLS)     # observe + reason + choose action
22        s.steps += 1
23        s.tokens += decision.usage.total_tokens
24 
25        if decision.final_text is not None:          # goal complete?
26            return decision.final_text
27 
28        call = (decision.tool_name, decision.tool_args)
29        if call == last_call:                        # stuck: same call twice in a row
30            raise RuntimeError("repeated tool call, aborting")
31        last_call = call
32 
33        handler = TOOLS.get(decision.tool_name)
34        if handler is None:
35            result = {"error": f"unknown tool {decision.tool_name}"}
36        else:
37            try:
38                result = handler(**decision.tool_args)   # use tool
39            except Exception as e:                      # receive result (even failure)
40                result = {"error": str(e)}
41 
42        s.messages.append(decision.as_message())        # update state
43        s.messages.append({"role": "tool", "name": decision.tool_name, "content": result})

Key points

  • Goal → observe → reason → choose action → use tool → receive result → update state → goal complete? → loop. Every agent is this.
  • The model chooses actions; the runtime owns the loop, the step counter, the budget and the tool dispatch.
  • State is everything known; context is what fits in this call. The gap between them is context engineering.
  • Termination needs multiple independent conditions enforced by code, not only the model saying "done".
  • Cost is roughly quadratic in steps because every iteration re-sends the growing context; measure per step.
  • Tool results are untrusted input and must be validated before they influence the next action.

The agent loop, step by step

Step through an agent execution
A support agent resolving a duplicate-charge refund. Watch which node of the loop is active and what the agent can see.
noyesGoalObserveReason / PlanChoose actionUse toolReceive resultUpdate stateGoal complete?Finish
Goal set. The user asks for a refund. The goal is the terminal condition the loop checks every iteration; without it the agent cannot decide when to stop.
Current goal
Resolve refund request for order #4821
Agent decision
—
Selected tool
—
Tool arguments
—
Tool result
—
Memory update
Loaded: customer prefers email replies (episodic memory).
Current state
{ goal: "Resolve refund request", iteration: 0, order: null, decision: null }
Context the model sees
system: You are a support agent. Use tools; never issue refunds above policy without approval.
user: I was charged twice for order #4821, please refund the duplicate.
Available tools
search_orders(customer_email)get_order(order_id)check_refund_policy(order)issue_refund(order_id, amount) [approval]reply_to_customer(text)
Tokens (in + out)0
Latency0 ms
Estimated cost0 ($0.0000)
1/15 · Goal set

When to use — and when not to

Use it when
  • Building or debugging any tool-using system — map the bug to a box in the loop.
  • Estimating cost and latency before committing to an agent design.
  • Deciding which parts to write yourself versus take from a framework (the loop itself is ~40 lines).
Avoid it when
  • The task is a fixed sequence — write the sequence, no loop needed.
  • A single call with structured output produces the result.
  • You cannot afford unbounded iteration and cannot define a step cap that still completes the task.

Failure modes

  • No step limit: a stuck agent burns budget until the provider rate-limits you.
  • Context overflow: state is appended forever, the window fills, and the model loses the original goal.
  • Tool exceptions propagate and kill the run instead of being returned to the model as an observation.
  • The same failing tool call is retried identically ten times because nothing detects repetition.
  • Per-step tokens are never recorded, so a 40x cost regression is discovered on the invoice.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

A hand-written loop is simple and inspectable; the ratings reflect one agent with a handful of tools.

Don't delegate understanding
The manifesto →
Don't delegate understanding.
← What Is Agentic AI?What Agentic Engineers Build →