Agent Observability Data
Prompt, model, tool calls, latency, tokens, outcome and feedback — as a high-cardinality event table with a classification and a retention policy, not as logs in a bucket.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What has to be recorded about an agent run for it to be explainable a week later, and how much of what makes it explainable is the customer's own words?
The on-call engineer reconstructing one bad answer; the evaluation sampler, whose entire raw material is this archive; product analytics counting resolutions; the team attributing token consumption to a feature or a tenant; and the compliance reviewer who has to say where a customer's message went and how long it stayed there (The Agent Returned 200 OK and the Answer Was Wrong).
There are two grains and both are needed. The storage grain is one span: one step of one run — a retrieval, a tool call, a model call — with its parent run id. The analytical grain is one run, assembled from spans. A conversation is a third unit above both. Metrics computed over spans and reported as runs are the standard counting error here (Grain: What Does One Row Represent?, Trace, Span, Attribute, Status).
Send traces wherever the existing observability stack sends logs, with default retention and default sampling, and get on with building the product. It is fast, it costs no design time, and for a prototype with no customers it is exactly right.
The archive contains verbatim customer messages, retained by whatever the platform default is, in a system whose access list was designed for application logs. The most sensitive dataset in the company was created by an observability tool (PII in Pipelines, Security-Safe Logging).
- The archive contains verbatim customer messages, retained by whatever the platform default is, in a system whose access list was designed for application logs. The most sensitive dataset in the company was created by an observability tool (PII in Pipelines, Security-Safe Logging).
- Run ids, tenant ids and prompt hashes are attached as metric labels. The metrics backend now stores a distinct series per run, and the cost and the query latency both grow with traffic rather than with the number of things being measured (Cardinality: The Label That Took Down Monitoring, Label Sets That Survive a Year).
- Sampling is head-based, so it keeps a fixed share of runs chosen before anything is known about them. The incident you are investigating has no trace and the ordinary successful runs have plenty (Sampling Without Throwing Away the Evidence).
- Token counts were never written beside a feature, a tenant or a request id, so consumption is one undifferentiated total with no dimension to group by and no way to attribute it (Cost Attribution).
- The prompt version, tool schema version and corpus version are not on the record. A regression that started on Tuesday cannot be attributed to any of the three things that changed on Tuesday (Semantic Changes).
- User feedback arrives hours later and is written as an update to the run row. Two updates race, one is lost, and the outcome column now disagrees with the feedback table (Late-Arriving Data, The Lost Update, Step by Step).
What is actually happening
- A trace archive is a high-cardinality, append-only event table with a retention policy — a fact table whose dimensions are the versions in play. Treating it as logs is what produces an archive nobody can query and nobody governs (The Event Log).
- Three destinations, three shapes, and mixing them up is the most common design failure. Metrics are low-cardinality aggregates and must never carry a run id. Traces are per-run, high-cardinality and sampled. The archive is the complete, long-retention record that evaluation and audit read (Logs, Metrics and Traces).
- The honest tension is unavoidable and worth stating plainly: the field that makes a trace useful for debugging is usually the customer's actual words. The exact message, the retrieved chunk text and the tool arguments are simultaneously the most diagnostic and the most sensitive content in the platform, and no policy setting is good at both ends (Data Classification). The resolution is tiering rather than choosing. Structure — ids, versions, lengths, token counts, tool names, outcome labels, hashes — answers most aggregate questions and contains almost no personal data, so it can be retained long. Content answers the "why did it say that" question and can be retained briefly (Data Retention, Data Minimization).
- The record is a fact table and the versions are its dimensions: prompt version, model version, tool schema version, corpus and embedding versions, tenant, feature, and the request id that joins everything. Without those columns the archive supports counting and nothing else (Fact Tables, Dimension Tables).
- Cost here means token and compute drivers, never currency: input tokens, output tokens, tokens spent on retrieved context, retries, tool calls, model class. Those are the columns that let anyone reason about consumption, and they are properties of a run rather than of a price list (Token Budgets).
- Feedback is late-arriving data in the strict sense this domain means: an event whose event time is the run and whose arrival time is hours or days later. Modelling it as an update to the run row invites lost updates; modelling it as a separate event stream joined on run id is the ordinary correct answer (Late-Arriving Data, Event vs Snapshot Modeling). The stream is at-least-once and unordered. A span can arrive twice and a later step can be written before an earlier one, so the run assembly has to be tolerant of both rather than assuming a sequence (At-Least-Once Delivery, Deduplication).
Eight things to record, and what each one costs in privacy
The list of fields is uncontroversial. What is usually missing is the third and fourth columns: what each field exposes, and how long it therefore ought to live. Deciding retention per field rather than per table is the design move that resolves most of the tension in this lesson (Data Retention).
Read the sensitivity column against the "answers" column and the trade becomes concrete. Almost everything that answers aggregate questions — how often, how many tokens, which tool, which version — carries little or no personal data. Almost everything that answers "why did it say that" is the customer's own words or the document they were shown.
That asymmetry is the design. Keep the structural columns for a long time because they are cheap and safe and answer most questions; keep the content columns briefly because they are large, sensitive, and only needed while an incident is live. A single retention policy over the whole record has to be wrong for one half or the other (Data Minimization).
| Recorded | What it answers | Sensitivity | Suggested tier |
|---|---|---|---|
| Prompt version and rendered prompt | Which instructions produced this behaviour; what changed on Tuesday (Semantic Changes). | The version is inert. The rendered prompt contains the user input and the retrieved context, and is therefore as sensitive as both. | Version: long. Rendered text: short, and only for sampled or failed runs. |
| Model and client library version | Whether a behaviour change coincides with a model or library change. | None. | Long. It is a few bytes and it makes regressions attributable. |
| Tool calls: name, arguments, result, error | What the agent actually did, and which step failed (Tool Errors, Retries and Timeouts). | Names and outcomes are inert. Arguments are where the account numbers are — the field most often missed by a redaction pass scoped to messages. | Names, latencies and outcomes: long. Arguments and results: short, redacted at capture where possible. |
| Latency per span | Where the time went, and which step is the bottleneck (Reading an Agent Run as a Trace). | None. | Long, and aggregate it early — per-span latencies compress well into summaries. |
| Token counts: input, context, output, retries | What drives consumption, per feature and per tenant (Token Budgets). | None. Counts are the safest useful column in the record. | Long. This is the column that makes attribution possible at all (Cost Attribution). |
| Retrieved chunk ids and text | Whether the answer was supported by what was retrieved — the single most useful debugging field (Citations). | Ids are inert and resolvable later. The text is a copy of corpus content and inherits the corpus classification exactly. | Ids and scores: long. Text: short, or omitted entirely if the chunk can be re-resolved from the id and corpus version. |
| Outcome label and error class | What happened, and the basis of every rate anyone reports. | Low, unless the label is free text — in which case it is unbounded and unreviewed. | Long, with a closed vocabulary. Free-text outcomes are how content leaks into a structural column. |
| User feedback | Whether the user thought it was right — the only ground truth that arrives on its own (Evaluation Data Pipelines). | Comment text is user-authored and can contain anything, including data the product never asked for. | Rating: long. Comment text: short and filtered, and never joined into an exported extract unfiltered. |
The pattern is stable across every row: identifiers, versions, counts and labels are small, safe and long-lived; text is large, sensitive and short-lived. A record designed that way answers aggregate questions for years and debugging questions for days, which is close to the actual shape of demand (Data Classification).
What makes a trace archive large
Trace storage is the cost that grows without anyone deciding it should, because it is written by production and deleted by nobody. The drivers are worth internalising as an ordering, because the largest one is also the easiest to reduce without losing anything you actually query (Storage Lifecycle).
The bars below are relative and unitless — they establish which lever matters, not a magnitude. The teaching is the ordering: content fields dominate everything else, so a tiered retention policy that expires text early and keeps structure is the single largest reduction available, and it costs almost no analytical capability.
Note where sampling sits. It is a large lever and it is the one to reach for last, because it trades away debuggability and evaluation material rather than bytes you were not using. Expiring content and pre-aggregating spans reduce size without reducing what you can ask (Sampling Without Throwing Away the Evidence).
The dominant term by a wide margin, and the one that shrinks most with a short retention tier while leaving every aggregate query intact.
A multiplier on everything. Applying one window to a record whose columns have wildly different value over time is the most common avoidable cost in an agent platform (Data Retention).
Multi-step agents with many tool calls produce many rows per run. Pre-aggregating into a run summary keeps the analytical path cheap without discarding the spans (Snapshot Tables).
Grows with product success rather than with anything you control, which is why the policy has to be set before it matters rather than after.
A large lever that trades debuggability and evaluation material for bytes. Reach for it after tiering, not before (Sampling Without Throwing Away the Evidence).
Small in a columnar table and disproportionate in a metrics backend, where a run id as a label creates a series per run (Cardinality: The Label That Took Down Monitoring).
Does not change bytes stored and changes bytes scanned by every evaluation sample and every incident query (Partitioning, Scan Cost).
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a typical multi-step agent archive, shown to establish an ordering rather than a magnitude. Two of the top three are retention decisions rather than volume facts, which is the useful conclusion: most trace archives are large because of a policy nobody set, not because of traffic.
How trace pipelines fail
Every row here has the same shape as a failure elsewhere in this domain — dropped rows, a cardinality blow-up, a lost update, a truncation — which is the argument for treating this archive as a dataset rather than as an operational concern of the observability stack.
The first row is the one that defines the discipline. A trace pipeline that sheds load under pressure discards exactly the traces from the period under pressure, which is when the traces are worth the most. Any sampling or shedding decision that is blind to outcome has this property (Sampling Without Throwing Away the Evidence).
The last row is the mitigation failing, and it is specific: a redaction pass written against the conversation looks thorough, passes review, and leaves the account numbers in the tool arguments untouched (PII in Pipelines).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Traffic spikes and the trace pipeline sheds load. | The incident under investigation has no traces; ordinary successful runs from the same hour have plenty. | Head-based sampling or backpressure dropping, both of which discard without regard to outcome (Sampling Without Throwing Away the Evidence). | Buffer durably between the application and the archive, shed by outcome rather than uniformly, and keep every error and every negative-feedback run regardless of ratio (A Dead-Letter Queue Is a Workflow, Not a Bin). |
| Run id or tenant id is added as a metric label. | Metrics queries slow across the whole platform and the metrics bill grows with traffic rather than with what is measured. | A high-cardinality dimension in a system that stores a series per distinct label value (Cardinality: The Label That Took Down Monitoring, Label Sets That Survive a Year). | Keep high-cardinality dimensions in the trace table where a query can filter them, and keep metric labels to a small closed vocabulary. |
| A record exceeds the transport or storage size limit. | Traces exist and the retrieved chunk text is cut off, reliably, in the longest and most interesting cases. | Silent truncation of the largest field rather than an explicit decision about what to keep. | Store chunk ids and resolve text from the corpus at read time, and record explicitly when a field was truncated rather than truncating silently (Citations). |
| User feedback arrives hours after the run. | The outcome column disagrees with the feedback table, and success rates for recent periods keep changing. | Feedback modelled as an update to the run row, so concurrent writes lose each other and there is no event history (The Lost Update, Step by Step). | Model feedback as a separate append-only stream keyed on run id, joined at query time, with the lag published so recent metrics are read as provisional (Late-Arriving Data, Stream Joins). |
| A new tool is deployed. | A new argument shape appears in the archive containing customer identifiers nobody classified. | Tool schemas evolve independently of the trace record's classification review (Data Classification). | Make classification of tool arguments part of adding a tool, and alert on unrecognised fields appearing in the record (Contract Enforcement). |
| A redaction pass is added after a privacy review. | The review passes, and account numbers are still present in the archive a month later. | The pass was scoped to the message body while tool arguments, retrieved text and error payloads flowed untouched — the mitigation failing. | Filter by field coverage rather than by field name: every field is either filtered or explicitly declared inert, and a new field defaults to filtered (Data Minimization). |
Feedback is late-arriving data
The one part of this archive that behaves like a stream problem is feedback, and it behaves like the canonical one: an event whose event time is the run and whose arrival time is much later. Every rule this domain teaches about late events applies without modification (Late-Arriving Data, Event Time).
The timeline below makes the consequence concrete. If a daily success rate is computed at the boundary and never revisited, the runs whose feedback arrived after the cut-off are counted as having none, and yesterday looks different from last week for a purely mechanical reason. If it is recomputed on every read, the number changes under people and nobody trusts it (Windows).
The workable answer is the ordinary one: publish the metric with an explicit allowed-lateness window, mark periods inside that window as provisional, and restate them once when the window closes. That is a decision about what the number means, made once, rather than an argument every month (Watermarks, Data Contracts).
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| run-a | 10:03 | 10:03 | Tuesday The run itself. Traces arrive within seconds and the record is complete except for feedback. |
| fb-a | 10:03 | 10:47 | Tuesday Feedback on run-a, same day, well before the boundary. Counted where everyone expects. |
| run-b | 22:41 | 22:41 | Tuesday A late-evening run. The trace is complete; the user has not responded yet. |
| fb-b | 22:41 | 09:15 | Tuesday, if lateness is allowed Feedback belongs to Tuesday by event time and arrived on Wednesday morning. Computed at midnight, Tuesday counts run-b as unrated. |
| run-c | 23:52 | 23:52 | Tuesday The last run of the day, and the one most likely to have its rating attributed to the wrong period. |
| fb-c | 23:52 | 14:30 | Tuesday, if lateness is allowed Arrived fourteen hours later. Attributing it to Wednesday by arrival time would move a rating onto a day whose run it did not describe (Processing Time). |
| fb-d | 11:20 | 11:05 | Tuesday, four days late A rating left after a support follow-up. Beyond any reasonable lateness window, and it will either be dropped or restate a figure people have already read (Late-Arriving Data). |
Times are clock labels on a teaching timeline, not measurements. The decision the timeline forces is what "Tuesday's success rate" means: computed at midnight it undercounts ratings; recomputed forever it never settles. Naming an allowed-lateness window and marking the provisional period is the only version that is both stable and honest.
How to build it
Most important first.
- Define the trace record as a schema with an owner and a contract, before the first field is emitted. It is a dataset with consumers — evaluation, analytics, cost, audit — and every one of them breaks when a field changes shape (Data Contracts).
- Store spans, and materialise a run-summary table beside them. Almost every analytical question is asked at the run grain, and re-deriving runs from spans on every query is the reason so many trace archives are unusable (Snapshot Tables).
- Write the versions on every record — prompt, model, tool schema, corpus, embedding, client library. They are what make a regression attributable, and they cannot be added retroactively to captured data (Data Lineage). Record token counts as columns on the run, broken down by input, retrieved context, output and retries, with tenant and feature beside them. That is what makes consumption groupable rather than a single total (Cost Attribution).
- Classify every field at design time and give content fields a shorter retention tier than structural ones. Two retention policies over one table is cheaper than one policy that is wrong for half the columns (Data Retention). Redact at capture where it is possible at all. A field never captured needs no filter, no audit and no deletion path — minimisation beats every downstream control (Data Minimization).
- Sample by outcome rather than uniformly: keep every error, every escalation, every low-confidence run and every case with negative feedback, and downsample the successes with a recorded ratio (Sampling Without Throwing Away the Evidence). Keep run ids, tenant ids and prompt hashes out of metric labels entirely. High-cardinality dimensions belong in the table, where a query can filter on them, not in a metrics backend that stores a series per distinct value (Cardinality: The Label That Took Down Monitoring).
- Model feedback as its own event stream keyed on run id, arriving late, joined rather than merged into the run row (Stream Joins). Put a lifecycle policy on the archive on the day it is created. Traces are written by production and deleted by nobody, which is the only cost driver in an agent platform that grows without anyone deciding it should (Storage Lifecycle).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The stream is at-least-once and unordered with respect to real time: a span can be written twice, and a later span can land before an earlier one from the same run (At-Least-Once Delivery).
- A run record is eventually complete, and there is no moment at which the platform can tell you it is. Assembling a run means deciding a cut-off, and a span arriving after the cut-off is a correctness problem you have chosen (Watermarks).
- Under sampling, the archive guarantees nothing about any individual run. "There is no trace" is not evidence that the run did not happen (Missing Rows).
- Feedback may never arrive for any given run, and its absence carries no information: a user who was satisfied and a user who left both produce silence (Nullability & Defaults).
- Token counts as reported by the serving path are what the platform observed. They are a property of the run as measured, and treating them as an authoritative external accounting is a claim the archive cannot support.
- Nothing here is rebuildable. This is captured data from production events that will not recur, so a week of dropped traces is a week of evaluation material and audit evidence gone permanently (Reprocessing vs Retrying).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check to build first is a drop-rate reconciliation: count runs served by the application, count runs present in the archive for the same period, and alert on the difference beyond the intended sampling ratio. It catches backpressure shedding, a collector restart, a misconfigured exporter and a sampler that changed behaviour (Reconciliation).
- It misses traces that arrived and are incomplete — a run whose tool spans were dropped, or whose content fields were truncated at a size limit. The count matches, the record is useless, and only a per-run span completeness check finds it (The Dimensions of Data Quality).
- Pair it with a field-population check per version of the record: the share of runs carrying each expected field. A field whose population rate falls to zero after a deployment is the fastest possible detection of a schema change nobody announced (Volume Anomalies).
- Two consumers, two freshness requirements, and one pipeline cannot serve both well. Incident debugging needs the last few minutes and tolerates incompleteness; evaluation and analytics need completeness and tolerate lag. Two paths over the same events is usually cheaper than one compromise (Lambda Architecture).
- Feedback lag is a property of humans and is not compressible. Any metric that joins feedback to runs is incomplete for a period after each run, and reporting it as final too early makes recent performance look different from older performance for purely mechanical reasons (Late-Arriving Data).
- Freshness of the archive should be measured as the age of the newest complete run, not as pipeline runtime. The distinction matters precisely during backpressure, when the pipeline is running perfectly and the data is hours behind (Freshness Monitoring).
- Adding a field is additive and applies only to runs captured after it. Captured data cannot be backfilled, so every field you did not think of is permanently absent from history — the strongest argument in this module for over-capturing structure while under-capturing content (Schema Evolution).
- New tools bring new argument shapes, which means new classification decisions. A tool added without a classification review is a new sensitive field flowing into an archive that was reviewed under different assumptions (Data Classification).
- Changing what an outcome label means — "resolved" moving from "the user did not come back" to "the user said yes" — is a semantic change with no schema footprint that invalidates every historical comparison (Semantic Changes, Two Dashboards, Two Numbers).
- The record is consumed by evaluation, analytics, cost attribution and audit simultaneously, so its schema deserves the deprecation discipline of a public interface rather than that of a log line (Backward Compatibility).
- There is no recovery for dropped traces. The only defence is upstream: a durable buffer between the application and the archive, with a dead-letter path for records that cannot be written, so backpressure produces delay rather than loss (A Dead-Letter Queue Is a Workflow, Not a Bin, The Event Log).
- Duplicate spans are handled by keying on span id and writing idempotently. This is one place where at-least-once delivery is genuinely harmless, provided the sink is keyed (Idempotent Data Pipelines, Deduplication).
- Reprocessing the archive — re-deriving run summaries, recomputing outcome labels, re-running a redaction pass — is an ordinary backfill over an immutable event table, and it is available only if the raw spans were kept (Backfills, Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- Backpressure shedding, so the traces missing from the archive are exactly the ones from the incident (Sampling Without Throwing Away the Evidence).
- Cardinality explosion from run or tenant ids used as metric labels, which degrades the metrics system for everything else it does (Cardinality: The Label That Took Down Monitoring).
- Content fields truncated at a size limit, so the retrieved chunk that explains the answer is cut off precisely in the interesting cases.
- Default retention, meaning forever, on the most sensitive dataset in the platform (Data Retention).
- Feedback written as an update to the run row, producing lost updates under concurrency and a column that disagrees with its own source (The Lost Update, Step by Step).
- The mitigation failing: a redaction pass applied to the message body while tool arguments, retrieved chunk text and error payloads flow through untouched.
- "Traces are logs." They are a dataset with a grain, a schema, dimensions, a freshness requirement and a classification, read by four different consumers. Treating them as log lines is why they end up unqueryable and ungoverned (The Event Log).
- "Sampling is a cost setting." Sampling decides what can ever be debugged, evaluated or audited from this period. It is a data-retention decision that happens to reduce cost (Sampling Without Throwing Away the Evidence).
- "We can add that field later." Traces are captured, not derived. A field added today is absent from all of history, and no backfill exists (Reprocessing vs Retrying).
- "Cost per run is a monetary figure." In this archive, cost is token and compute drivers — inputs, outputs, retries, tool calls. Those transfer between providers and across time; a recorded price does not (Token Budgets).
- "Personal data is in the message body." It is in tool arguments, retrieved chunk text, error payloads, structured metadata and free-text notes. A redaction pass scoped to the conversation misses most of it (PII in Pipelines).
- "The dashboard shows the success rate." It shows the success rate of the runs that were sampled and whose feedback has arrived so far. Both qualifiers move the number, and neither appears on the chart (Late-Arriving Data).
- This is usually the most sensitive dataset in an agent platform and usually the least governed, because it was created by an observability tool rather than by a data pipeline. Classification, access control and retention belong on it from day one (Data Classification, Data Access Control).
- Personal data in *metrics* is worse than in traces: metrics are retained longer, read more widely and exported to more places, and a label value cannot be redacted after the fact (What You Just Wrote Into a Log Half the Company Can Read).
- Deletion requests must reach spans, run summaries, any derived evaluation cases and any exported extract. That is only tractable if tenant and subject identifiers are columns rather than substrings of free text (Deletion Requests).
- Tiered retention is itself a governance control: content expiring quickly while structure persists means most of the archive stops being personal data long before it stops being useful (Data Retention, Data Minimization).
Operating it
- Trace drop rate and trace ingestion lag, reported separately. They have different causes and the same symptom, and conflating them means every incident starts with the wrong hypothesis (The Backlog Arithmetic: Four Levers and a Drain Time).
- Spans per run distribution, with both tails inspected. A run with one span is an assembly failure; a run with hundreds is usually a loop that did not terminate (Distribution Tests).
- Share of runs carrying each expected field, per record version, so a deployment that stopped emitting something is visible the same day (Pipeline Metrics). Cardinality of every dimension actually used as a label or partition key, monitored rather than assumed (Label Sets That Survive a Year).
- Bytes retained per tier — structural versus content — because the two grow at different rates and only one of them is a privacy liability (Storage Lifecycle).
- Token counts grouped by feature and tenant, which is the whole reason those columns exist (Cost Attribution).
- Share of runs with feedback attached, and the lag distribution of that feedback, so metrics that depend on it are read with the right amount of confidence (Late-Arriving Data).
- Trace volume scales with usage while corpus volume scales with content, so the two curves cross early and never cross back. Plan the lifecycle policy for the trace archive first (Storage Lifecycle).
- At ten times, the run-summary table stops being a convenience and becomes the only affordable way to answer analytical questions, because re-deriving runs from spans per query no longer fits (Snapshot Tables).
- At a hundred times, tiering becomes mandatory in two dimensions at once: content retained briefly and structure retained long, and hot storage for recent data with cold storage for the rest (Storage Lifecycle, Raw, Staging, Curated: Layers by Purpose).
- Tenant count scales the governance problem faster than volume scales the storage one. Per-tenant retention and deletion obligations turn one policy into many, and they must be expressible as predicates over columns that were captured (Multi-Tenant Isolation).
- Retained bytes dominate, and they grow with usage rather than with content — which is why the trace archive overtakes the retrieval corpus early and then keeps going. It is the first storage surprise in most agent platforms (What Actually Drives Data Platform Cost).
- Content fields are most of those bytes. Messages, retrieved chunk text and tool payloads are large; ids, versions, counts and labels are small, which is what makes tiered retention such an effective lever (Data Minimization).
- High-cardinality dimensions are cheap in a columnar table and ruinous in a metrics backend. The same field costs almost nothing in one place and dominates the other (Cardinality: The Label That Took Down Monitoring).
- Evaluation sampling scans the archive on a schedule, so layout matters: partitioning by day and clustering by tenant or feature is what keeps that scan proportional to the period being sampled (Partitioning, Scan Cost).
- Token counts recorded on the run are the driver to reason about consumption — input, retrieved context, output, retries. They are what makes a heavy feature identifiable; they are not a price and this domain never records one (Token Budgets).
- Full-fidelity traces are the best debugging asset and the largest privacy liability the platform holds. Every retention and redaction choice moves along that single axis and there is no setting that is good at both ends — the tiering design is an attempt to be on two points of the axis at once, for different columns.
- Outcome-based sampling keeps what matters and destroys any claim that the archive is a representative sample of production. Record the sampling ratios, or every downstream rate computed from it is wrong by an unknown factor.
- Capturing structure generously costs storage on every run forever and buys the ability to answer questions nobody has asked yet — which for captured, non-rebuildable data is the only chance you get.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe record shape, the three destinations, the tiering of structure against content and the late-arriving nature of feedback hold for any agent stack and any tracing tool. What differs is which system holds each destination and how much of the schema the tool imposes, not which columns have to exist for a run to be explainable.
- ORG-SPECIFICHow long content fields may be retained, whether verbatim customer messages may be stored at all, and who may read the archive are decided by classification policy and regulation rather than by engineering. An internal developer tool and a consumer support product can share every line of pipeline code and have completely different retention tiers.
- SCALE-SPECIFICBelow a modest traffic volume, keeping every trace at full fidelity for a short window is simpler and more useful than any sampling design, and a run-summary table is unnecessary. Sampling, tiering and pre-aggregation become mandatory at the point where the archive stops fitting a scan for evaluation, which arrives from usage growth rather than from content growth.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the delivery semantics this stream inherits — why a span arrives twice, why a later step is written before an earlier one, and what a replay of the archive actually replays.
- — DevOps / Production Engineering owns the operational side of the same data: what pages a human, what an error budget is computed from, and how a change to the trace schema is rolled out across a fleet without a gap in the record.