Tracing Agents

• Input: for an LLM span, the exact messages sent — system prompt version, retrieved context, tool definitions, prior turns. For a tool span, the parsed arguments. Store by reference (content hash) when large.
• Output: the raw model response including any tool-call request; the tool's return value including error payloads.
• Tokens: prompt, completion and cached tokens per LLM span; the running total on the parent.
• Cost: computed at span end from tokens × the model's price at the time, plus tool charges. Never recompute historical cost from today's prices.
• Latency: start and end timestamps; time-to-first-token for streaming calls.
• Errors and retries: exception type, status code, attempt number, backoff applied; each retry is its own child span so five 429s are visible as five spans, not one slow one.
• State changes: which memory or scratchpad keys were written, which approval gate was passed, which branch a router chose. Without these, trajectory checks in Deterministic Evaluators have nothing to read.
• Identity: trace_id, span_id, parent_span_id, session_id, user_id (hashed), model id, prompt version, dataset case id when running evals.

1import time, uuid, contextvars
2 
3_current = contextvars.ContextVar("span", default=None)
4spans: list[dict] = []
5 
6class span:
7    def __init__(self, name, kind, **attrs):
8        self.s = {"span_id": uuid.uuid4().hex, "parent_span_id": None,
9                  "name": name, "kind": kind, "attrs": attrs}
10    def __enter__(self):
11        parent = _current.get()
12        self.s["parent_span_id"] = parent["span_id"] if parent else None
13        self.s["start_ms"] = time.time() * 1000
14        self.token = _current.set(self.s)
15        return self.s
16    def __exit__(self, exc_type, exc, tb):
17        self.s["end_ms"] = time.time() * 1000
18        if exc: self.s["error"] = f"{exc_type.__name__}: {exc}"
19        _current.reset(self.token)
20        spans.append(self.s)
21 
22def call_model(client, messages, tools, prompt_version):
23    with span("llm.call", "llm", prompt_version=prompt_version, n_messages=len(messages)) as s:
24        r = client.chat(messages=messages, tools=tools)
25        s["attrs"].update(input=messages, output=r.raw,
26                          tokens_in=r.usage.prompt, tokens_out=r.usage.completion,
27                          cost_usd=price(r.model, r.usage))
28        return r
29 
30def call_tool(registry, name, args):
31    with span(f"tool.{name}", "tool", input=args) as s:
32        out = registry[name](**args)
33        s["attrs"]["output"] = out
34        return out