Observabilityobservabilityloggingmetricsalertingdashboards

Logging, Metrics and Alerts

Structured logs, RED metrics plus agent-specific ones (steps, tokens, cost per task), alerts on loops and cost anomalies, and dashboards — with PII redacted before anything leaves the process.

Interview question
Progress

Structured logs

Log as JSON, one event per line, with the same identifiers as the trace: trace_id, span_id, session_id, user_id (hashed), model, prompt_version. Free-text log lines are unsearchable at the volume an agent produces; structured fields let you answer "show me every run where tool=issue_refund and status=error in the last hour" in one query. Log events at decision points — tool chosen, guardrail triggered, approval requested, retry attempted, budget exceeded — rather than every internal function call; the trace already has the fine grain.

Keep logs and traces distinct in purpose: traces are for reconstructing one run, logs are for searching across runs. Both share ids so you can jump between them.

One structured event with the trace ids attached and payload redacted before emission.
1import json, re, time
2 
3PII = [(re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "<email>"),
4       (re.compile(r"\b(?:\d[ -]?){13,19}\b"), "<card>"),
5       (re.compile(r"\+?\d[\d ()-]{8,}\d"), "<phone>")]
6 
7def redact(text: str) -> str:
8    for pattern, tag in PII:
9        text = pattern.sub(tag, text)
10    return text
11 
12def log_event(event: str, ctx: dict, **fields):
13    payload = {"ts": time.time(), "event": event,
14               "trace_id": ctx["trace_id"], "span_id": ctx["span_id"],
15               "session_id": ctx["session_id"], "prompt_version": ctx["prompt_version"]}
16    for k, v in fields.items():
17        payload[k] = redact(v) if isinstance(v, str) else v
18    print(json.dumps(payload))
19 
20log_event("tool.selected", ctx, tool="issue_refund", step=3, args_valid=True)
21log_event("budget.exceeded", ctx, kind="steps", limit=8, actual=9)

RED metrics and agent metrics

The RED method — Rate, Errors, Duration — is the standard starting point for any service. For an agent, record them per run and per tool: requests per minute, error rate by error type (tool failure, model 429, validation rejection, guardrail block), and duration as p50/p95/p99. Then add the metrics that only agents have: steps per run (distribution, not mean), tokens per run split by prompt/completion, cost per task, tool-call counts by tool, retry counts, guardrail triggers, approval-gate outcomes and cache hit rate.

Tag every metric with model, prompt_version and the agent or route name. When a model upgrade doubles p95 latency, the tag is what makes the dashboard show it in one panel instead of hiding it in a global average.

  • Rate: runs/min, tool calls/min, LLM calls/min.
  • Errors: per type — provider errors, tool errors, schema-validation failures, guardrail blocks, step-cap hits.
  • Duration: run latency p50/p95/p99; LLM call latency; tool latency per tool; time-to-first-token.
  • Agent-specific: steps/run, tokens/run, cost/task, retries/run, cache hit rate, escalations to human.

Alerting on loops, step counts and cost

Alerts should fire on the failure modes specific to agents, not just on HTTP 5xx. A run that hits the step cap is an alert-worthy event on its own — the cap is a safety net, not a normal exit (Budgets, Limits and Termination). A rising p95 step count with a flat success rate means the agent is flailing. Cost per task above a threshold, or total spend per hour deviating from the same hour last week by more than a set factor, catches runaway loops and prompt-size regressions early enough to matter (cost-explosion-after-launch). Repeated identical tool calls within a run is the loop signature and can be detected in-process and alerted immediately.

Set thresholds from the eval suite and the first weeks of production: p95 steps ≤ 1.5× the eval p95, cost per task ≤ 2× the eval median, zero step-cap hits per hour as the ideal and a low count as the page threshold. Route budget and loop alerts to whoever can roll back a prompt, because that is usually the fix.

  • Page on: step-cap hit rate, cost/hour anomaly, safety-guardrail block rate spike, provider error rate.
  • Ticket on: p95 latency regression, rising retry counts, cache hit rate drop, negative feedback rate.
  • Kill-switch: an automatic fallback to a cheaper model or a paused route when hourly spend crosses a hard limit (Fallbacks, Caching and Model Routing).

Dashboards and PII

A useful agent dashboard fits on one screen: success/escalation rate, p50/p95 latency, p95 steps, cost per task and total spend, error rate by type, tool-call volume by tool, and the sampled-trace quality score from Regression Gates and Online Evaluation — all split by prompt_version and model. Add a table of the most recent step-cap hits and guardrail blocks linking to their traces, so an on-call engineer goes from alert to Trace Inspection: Debugging from a Trace in one click.

PII redaction must happen before data leaves the process, not in the backend. Prompts and tool outputs carry names, addresses, emails, card numbers and free text that may contain anything. Redact known patterns with regexes, hash user identifiers, and for high-sensitivity domains store payloads by reference in a store with its own access control and retention while the trace holds only the hash. The trade-off is real — redacted context makes trace inspection harder — so keep an audited path for authorised engineers to retrieve the original for a specific trace_id.

  • Redact in-process; assume the logging backend is less trusted than the application.
  • Hash user ids with a keyed hash so joins still work but the raw id is not in logs.
  • Retention: full payloads days to weeks, aggregates indefinitely; document it.
  • Never log secrets, API keys or full tool credentials — including inside model inputs (see Secrets and Untrusted Output).

Key points

  • JSON logs with trace ids at decision points; traces for one run, logs for search across runs.
  • RED metrics (rate, errors, duration) plus steps, tokens, cost per task, retries, guardrail triggers.
  • Tag every metric with model and prompt version so regressions are attributable.
  • Alert on step-cap hits, rising p95 steps, cost anomalies and repeated identical tool calls — not only on 5xx.
  • A one-screen dashboard split by prompt version, linking alerts to traces.
  • Redact PII in-process before emission; hash identifiers; keep an audited retrieval path.

When to use — and when not to

Use it when
  • Before the first production deployment of any agent with a per-run cost.
  • When adding a new tool or route — extend the metrics and alerts at the same time.
  • After a cost or loop incident, to make the next one visible within minutes.
Avoid it when
  • Do not log every internal call as an event; the trace has the fine grain and logs become noise.
  • Do not rely on the provider's billing page as your cost alert; it lags by hours to days.
  • Do not ship raw prompts and tool outputs to a third-party log service without redaction.

Failure modes

  • Cost alert missing; a looping agent runs all weekend and the invoice is the first signal.
  • Metrics not tagged by prompt version; a regression is averaged into the global p95 and invisible.
  • Step-cap hits treated as successful completions; the agent "finishes" 10% of runs by timeout.
  • PII redacted in the backend after the raw payload has already been written to disk and shipped.
  • Dashboard shows mean latency and mean steps; the p95 that users feel is hidden.
← Trace Inspection: Debugging from a Trace