AgenticGENERALSCALE-SPECIFICCLOUD-SPECIFIC

Agent Audit Logs

What was asked, what the agent decided, which tools ran with which arguments, and what changed as a result.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

A customer says the assistant refunded the wrong order. What record lets you answer what happened, and why?

The requirement

Support, security and engineering all need to reconstruct an agent run after the fact — and the finance team needs to know which changes were made by a human and which by an assistant.

The obvious build

Log the user's message and the final answer. The tools write their own logs already, and the model provider keeps a copy of the conversation.

Why it breaks

The final answer is prose. It says a refund was issued; it does not say which order, which amount, or under whose authority (Agent Authorization).

How it breaks in production

The final answer is prose. It says a refund was issued; it does not say which order, which amount, or under whose authority (Agent Authorization).
Tool logs exist but are not correlated with the run, so reconstructing a trajectory means joining timestamps by hand across services (Correlation Ids That Survive Every Hop).
Nondeterminism means you cannot re-run the request to find out what happened — the record is the only evidence there will ever be.
Denied and failed tool calls are usually not logged at all, and those are exactly the ones a security investigation needs.
Provider-side conversation history is not an audit log: it is outside your retention control, outside your access control, and contains no record of what your database actually did.
When the same operation can be performed by a human or by an agent, an undifferentiated log makes "was this automated" unanswerable.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Two different records are needed and they are routinely conflated. The trajectory is a debugging artefact: prompts, model outputs, tool calls, results, timings, cost. The audit record is a compliance and support artefact: who authorised what, which action ran, what changed.
They differ in retention, access and sensitivity. Trajectories are large, contain full prompt content, and can be sampled and expired quickly. Audit records are small, must be complete, and are usually retained far longer (Audit Logs for Privileged Actions in the Security domain).
The unit is the run, identified once and attached to every model call, tool call, log line and database write it produces (Correlation Ids That Survive Every Hop).
The delegation chain must be explicit on every entry: the human, the run, the tool, the credential used. "The assistant did it" is not an actor (Agent Authorization).
Arguments are the substance of an agent audit log, because the arguments — not the tool name — are what the model chose. refund_order tells you nothing; refund_order(order_9f2, 4200) tells you everything.
The record must state what changed, not only what was attempted: entity, field, before and after. A tool call that failed halfway leaves the two questions with different answers.

Two records, not one

The first design decision is recognising that debugging and auditing want different things. Conflating them produces either an audit trail that is sampled away or an observability pipeline retained for seven years at enormous cost and risk.

Property	Trajectory (debugging)	Audit record (accountability)
Question it answers	Why did the agent do that?	What was done, by whom, to what?
Contents	Prompts, model outputs, tool results, timings, tokens, cost	Actor, run id, tool, arguments, target, before/after, outcome
Size	Large, grows with tokens	Small, grows with actions
Sampling	Yes — but never for errors or denials	Never
Retention	Days to weeks	Months to years, per policy
Mutability	Ordinary log storage	Append-only, tamper-resistant
Access	Engineering	Privileged; often per-tenant readable
If it is missing	A bug you cannot explain	A compliance and support failure

What an audit record has to contain

SIMPLIFIEDField names are illustrative. What matters is that actor, authority, arguments, effect and a pointer to the trajectory are all present in one record and queryable individually.

The test is simple: could someone who was not present reconstruct the action, the authority behind it and its effect, using only this record? Anything less leaves an investigation dependent on memory.

One tool invocation, recorded

1{
2  "ts": "2026-03-04T09:14:22.481Z",
3  "run_id": "run_01HW3K9Q2M",
4  "step": 4,
5 
6  "actor": {
7    "human_id": "u_8812",
8    "human_role": "support_agent",
9    "tenant_id": "t_204",
10    "on_behalf_of": "cust_5511",
11    "via": "assistant",
12    "credential": "role:support_scoped"
13  },
14 
15  "tool": "refund_order",
16  "arguments": { "orderId": "ord_9f2", "amountCents": 4200, "reason": "damaged" },
17  "authorization": { "decision": "allow", "policy": "refund.support.v3" },
18 
19  "outcome": "succeeded",
20  "duration_ms": 412,
21  "idempotency_key": "refund:ord_9f2:4200",
22 
23  "changed": [
24    { "entity": "order", "id": "ord_9f2",
25      "field": "refunded_cents", "before": 0, "after": 4200 },
26    { "entity": "refund", "id": "rfnd_77c", "created": true }
27  ],
28 
29  "trajectory_ref": "trace_4b1c9e",
30  "user_message_hash": "sha256:9c1f...",
31  "redactions": ["user_message"]
32}

Four things make this useful and are the ones most often missing: on_behalf_of and credential (the delegation chain), the authorization decision including the policy version, changed recorded from the write path rather than from intent, and trajectory_ref so the debugging record can be found without embedding it here.

Where audit trails go wrong

Each of these produces a log that looks complete on a normal day and is useless in the incident it exists for. They are worth checking against an existing implementation rather than discovering during an investigation.

Audit failures and their fixes

Trigger	Symptom	Cause	Response
Only successful calls are recorded	An attempted privilege escalation leaves no trace	Logging placed after the authorization check	Record the invocation and its decision before executing (Agent Authorization)
No run id	Tool logs cannot be assembled into a trajectory	Correlation identifier never created or never propagated	Generate at entry; propagate through model calls, tools and writes (Correlation Ids That Survive Every Hop)
Audit written before commit	Records of changes that were rolled back	Write outside the transaction boundary	Same transaction, or an outbox with the change (The Transactional Outbox)
Prompts logged in full	Log store becomes the most sensitive dataset in the company	No redaction at write time	Redact and hash on the way in; keep the hash for correlation (Secrets in Logs)
Uniform sampling across all runs	The one run under investigation was not kept	Sampling policy applied to audit as well as trajectory	Never sample audit; force-keep trajectories for errors and denials
Intent recorded instead of effect	"Refund issued" logged for a refund that partially failed	Audit written by the caller, not by the write path	Record before/after from the transaction that made the change
Human and agent actions indistinguishable	Cannot answer "how many refunds were automated"	No `via` or actor-type field	Mark the channel on every action, agent or not
Audit store writable by the service	A compromised service can rewrite its history	Same credentials for application and audit storage	Append-only storage with a separate, write-only credential (Defence in Depth)

How to build it

Most important first.

Generate a run id at the entry point and propagate it through the model calls, every tool call and every write (Request Context Propagation).
Record every tool invocation, including denied and failed ones, with tool name, arguments, outcome and duration.
Record the delegation chain: authenticated human, agent run, tool, and the credential the tool used.
Record what changed, from the write path rather than from the tool's intent — ideally as a domain event emitted by the same transaction (The Transactional Outbox).
Redact at write time. Prompts and tool results contain personal data, tokens and payment details; masking on read is a leak waiting for a misconfiguration (Secrets in Logs).
Separate the two stores. Trajectories in the observability pipeline with short retention and sampling; audit records in durable, append-only storage with long retention (Structured Logging).
Make audit records append-only and immutable, and treat access to them as privileged.
Log the reason for stopping — answered, budget, denial, error — so the run is interpretable without reading the whole trajectory (Budgets, Deadlines and Step Limits).
Emit a trace alongside the log, with a span per model and tool call, so latency and cost are attributable to steps (Tracing From the Backend's Side).

What can go wrong

Failure modes

Logging only successful tool calls, which removes exactly the evidence a security investigation needs.
Full prompt logging that quietly becomes the largest and most sensitive dataset the company holds (Secrets in Logs).
Sampling applied uniformly, so the rare interesting run is the one that was dropped. Sample trajectories; never sample audit records.
Run ids generated per tool call rather than per run, making correlation impossible.
Audit written by the tool before the transaction commits, so a rolled-back change is recorded as having happened (Where the Transaction Boundary Goes).
Arguments logged as a JSON blob with no schema, so the field that matters cannot be queried during an incident.
Trajectories stored with the model provider only, leaving no record when the provider relationship ends.

What can race

Audit written outside the transaction can record a change that was rolled back, or miss one that committed — write it in the same transaction or through an outbox (The Transactional Outbox).
Concurrent runs for one user interleave in the log; without a run id their entries are indistinguishable (Backend Races).
A run cancelled mid-tool can produce a side effect after the run's final record, leaving the audit trail out of order unless the tool writes its own entry.

Security

Audit logs are the primary evidence for privilege-escalation and prompt-injection investigations; without denied-call records, an attempted attack is indistinguishable from silence (Agent Authorization).
The logs themselves are sensitive: prompts contain personal data, tool arguments contain identifiers, and tool results may contain secrets. Access to the audit store is a privileged capability (Least Privilege in the Security domain).
Tamper resistance matters — append-only storage, or a separate account, so a compromised service cannot rewrite its own history.
Retention is a legal question as much as a technical one, and prompts routinely contain data subject to deletion requests. Design deletion into the trajectory store from the start.
Never log raw credentials, tokens or full payment details, even when the model emitted them into an argument (Secrets Are Not Configuration).

Misreads

"The model provider stores the conversation." Their retention, their access control, their jurisdiction — and no record of what your database did (The Trust Boundary).
"We log the request and the response." The interesting part is between them, and it is not reconstructible from either.
"Tool logs are enough." Without a shared run id, they are a pile of events with adjacent timestamps (Correlation Ids That Survive Every Hop).
"Audit and observability are the same pipeline." They have different retention, access, completeness and cost profiles; sampling one is normal and sampling the other is a compliance failure.
"We can re-run it to see what happened." The system is nondeterministic. The record is the only evidence.

Operating it

How you see it in production

Runs per user and per tenant, with termination reason, from the audit store rather than from application logs.
Tool invocation counts split by outcome: succeeded, denied, validation-failed, errored, timed out.
Time and cost attributable per step from the trace, so an expensive run can be explained (Budgets, Deadlines and Step Limits).
Audit write failures as their own alert — a missing audit record is a compliance incident even when the action succeeded.
Volume and cardinality of trajectory logging, because it grows faster than anyone expects (The Log Bill and What It Is Buying in the Observability domain).

What changes at 10x and 100x

Trajectory volume grows with tokens, not with requests, so it outpaces conventional logging quickly. Sample aggressively and always keep errors and denials.
Audit volume grows with actions and is small by comparison; keep all of it.
At multi-tenant scale, audit records need per-tenant access so a customer can be shown their own history without exposing anyone else's (Tenant Isolation).
At small scale, one structured log with a run id covers both needs — the split becomes necessary when retention or access requirements diverge.

What this costs

Complete trajectory logging is what makes agents debuggable and is expensive and privacy-sensitive. The compromise is sampling plus guaranteed retention of failures.
Redaction at write time protects the pipeline and destroys information you may later want.
Immutable audit storage is harder to correct when something is written wrongly — which is the intended property.
Recording arguments in full is essential for investigation and is precisely what makes the store sensitive.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALIndependent of provider and framework; what varies is only where the trajectory is captured.
SCALE-SPECIFICOne structured log with a run id is sufficient for a single-team internal tool. Separate trajectory and audit stores become necessary once retention, access control or regulatory requirements differ between the two.
CLOUD-SPECIFICAppend-only, tamper-resistant storage is provided differently by each platform — object versioning with retention locks, a managed audit service, or a separate account with write-only access. The requirement is the same; the mechanism and its guarantees are not identical across providers.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designidempotency-keys

Architecturedistributed-tracing event-sourcing

Securityaudit-logs logging-security least-privilege

Performanceagent-observability agent-trace-performance trace-anatomy log-cost-and-sampling structured-logging

Agentictracing-agents logging-metrics-alerts trace-inspection agent-to-agent-communication

Domains that do not exist yet

— Testing & Reliability Engineering — replaying a recorded trajectory as a regression test, and using stored runs as the golden dataset for evaluation.