AI DataGENERALSCALE-SPECIFICORG-SPECIFIC

Data Engineering for Agents

An agent system is a data platform with a model in the middle: six datasets, each with a grain, an owner, a freshness target, and its own quiet way of rotting.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

An agent gave a customer the wrong answer on Tuesday. Which datasets did it read, which did it write, and who owns each one?

Who needs this

The retrieval layer that reads the corpus on every request; the evaluation harness that reads traces and writes verdicts; the on-call engineer reconstructing one bad answer; the product team counting resolutions; the finance team attributing token spend to a feature; and the compliance reviewer who has to say where a customer's message went (The Agent Returned 200 OK and the Answer Was Wrong, What One Agent Run Costs, and Which Term Dominates).

What one row is

There is no single grain here, and noticing that is the lesson. One source document, one extraction, one chunk, one vector, one conversation, one turn, one tool call, one evaluation case, one feedback event — nine different units, routinely stored in systems that were never asked to declare which one they hold (Grain: What Does One Row Represent?).

The obvious build

A notebook loads a folder of documents into a vector store, the prompt lives in the application repo, and traces go wherever the observability stack already sends logs. Nothing is scheduled, nothing is versioned, and for the first demo that is exactly right — building ingestion infrastructure before you know whether retrieval helps at all is how six months disappear.

Why it breaks

A policy document is revised. Nothing re-ingests it, so the agent keeps quoting a paragraph that no longer exists and attaches a citation that looks entirely credible (Citations).

How it breaks with real data

A policy document is revised. Nothing re-ingests it, so the agent keeps quoting a paragraph that no longer exists and attaches a citation that looks entirely credible (Citations).
A document is deleted for a legal reason. The row is gone from the source system and its vectors are still in the index, which means the deletion was not honoured and no query anyone runs will reveal that (Deletion Requests).
Someone changes the chunk size to fix one bad answer. The index now holds two chunking strategies at once, no column records which is which, and retrieval quality has stopped being attributable to anything (Chunking Pipelines).
The evaluation set is a spreadsheet one engineer curated in March. It has no grain, no version and no provenance, and a third of its expected answers contradict a policy that changed in June (Golden Datasets).
Traces land in a bucket with default retention and no classification. They contain verbatim customer messages, which makes the debugging archive the most sensitive dataset in the company and the only one with no owner (PII in Pipelines).
Spend appears as one line with no dimension to group by, because token counts were never written next to a feature, a tenant or a request id (Cost Attribution).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Take the model out of the picture and a familiar shape is left: a batch pipeline that builds a derived index from source documents, a serving path that queries that index, an append-only stream of what happened, and offline jobs that read the stream. The parts have new names; the failure classes are the ones this domain has been describing all along (The Data Loop).
Two of these datasets are derived. The chunk table and the vector index are functions of source documents, a chunking strategy and an embedding model version. Derived means rebuildable, and rebuildable means both the inputs and the function have to be recorded — which is precisely what almost nobody does (Data Lineage).
Two are captured. Traces and feedback are event streams produced by production: they arrive late, out of order and at least once, with every property this domain already teaches about event data (The Event Log, Late-Arriving Data).
One is curated. An evaluation set's entire value is its provenance; a set whose origin you cannot explain is a number generator with good manners (Evaluation Data Pipelines).
One is operational. Prompts, tool schemas and retrieval configuration are configuration that changes the meaning of every dataset downstream of them without changing a single schema (Semantic Changes).
These rot faster than an ordinary warehouse table because no consumer complains. A stale fact table produces a number a finance analyst disputes on sight; a stale chunk produces a fluent, confident, wrong sentence that the reader has no independent way to check (The Pipeline Succeeded. The Data Is Wrong.).

Six datasets, and what one row of each represents

Ask where the data is in an agent system and the usual answer is "the vector database". That is one of at least six datasets, and it is not the one that causes the most incidents. The table below is the inventory; most of the rest of this module is a lesson per row.

The column doing the real work is the second one. Every argument about retrieval quality that goes in circles is an argument between two people holding different units — one talking about documents, the other about chunks — and writing the grain down ends it in about a minute (Grain: What Does One Row Represent?).

Read the last column twice. Each of these rots in a way that leaves the system running, the job green and the answer fluent. There is no row where the failure announces itself, which is why every one of them needs an explicit check rather than an absence of alerts.

Dataset	One row is	Kind	Read by	Rots when
Source documents	One document at one version, as its owner published it.	Authored	The ingestion pipeline, and nothing else.	Someone edits it and the pipeline is not told, so every derived row below is quietly describing a version that no longer exists.
Extraction output	One document rendered as text plus structure, with the extractor version recorded.	Derived	The chunker.	A parser upgrade changes how tables and headings come out, and the change is invisible because the row count barely moves.
Chunk table	One retrievable unit of text with its offsets, its parent document and its chunking strategy.	Derived	The embedding job, and the retriever for its text and metadata.	The strategy changes and old chunks stay, so the index holds two grains at once and neither is labelled.
Vector index	One vector for one chunk under one embedding model version.	Derived	The retrieval path, on every single request.	The embedding model changes and only part of the corpus is recomputed, so distances are compared across two spaces (Re-embedding).
Trace archive	One agent run, or one span within it: prompt, model, tool calls, tokens, latency, outcome.	Captured	On-call engineers, the evaluation sampler, cost attribution, and any audit.	Sampling silently drops the interesting tail, or retention expires before anyone asks the question the traces would have answered.
Evaluation set	One case: an input, the context it should retrieve, and what a correct response looks like.	Curated	The eval runner on every change to prompt, model, corpus or chunker.	The world changes and the expected answers do not, so the suite passes while production gets worse (Golden Datasets).

Three kinds, three different recovery stories. Derived datasets are rebuildable if you kept their inputs and versions. Captured datasets are not rebuildable at all. Curated datasets are rebuildable only in the sense that a human could do the work again.

The flows, drawn

SIMPLIFIEDDrawn as one corpus and one index. Real deployments run an index per tenant or per document class, often several retrievers whose results are merged, and a build loop that is incremental rather than the single scheduled sweep this shows — each of which adds a place where one part of the corpus can be at a different version from the rest.

Two loops and one archive. The build loop runs offline and turns documents into an index; the serving loop runs per request and turns a question into an answer; the archive is written by the serving loop and read by everything that wants to know how it went.

Notice which edges are scheduled and which are per-request. The build path is a batch pipeline with all the batch questions attached to it — windowing, incrementality, idempotency, atomic publish. The serving path is a low-latency read against a derived index. Confusing the two is how people end up trying to make ingestion "real-time" when the actual requirement was that a document edited this morning be retrievable this afternoon (Batch vs Streaming Ingestion).

The dashed relationship worth internalising is the one from embedding model version to both the index and the query encoder. A query is embedded at request time by some model, and the vectors it is compared against were embedded at build time by some model. If those two are not the same version, retrieval is not degraded — it is meaningless, and it still returns ten confident results (Re-embedding).

Build loop, serving loop, and the archive that connects them

UserLLMAgentToolDataDecisionHumanGuardrail

How each dataset rots, and what the symptom looks like

The reason to enumerate these is that the symptom is nearly always the same — "the agent gave a bad answer" — while the cause is in one of six different systems owned by four different people. Without a table like this, every investigation starts by re-deriving it under pressure.

The pattern to notice in the cause column: not one of these is a model failure. Every row is an ordinary data-engineering fault — a missed change, a partial migration, a lost event, a stale curated set, a broken derivation — and every one of them would be routine to detect if the artefact had been treated as a dataset from the start.

The response column is deliberately boring. Coverage reconciliation, version columns, atomic publish, retention policy, freshness monitoring: this domain already owns all of them, and the work is applying them to nouns that arrived recently rather than inventing anything.

Symptom to cause across the six datasets

Trigger	Symptom	Cause	Response
A source document is edited or deleted.	The agent quotes text that no longer exists, with a plausible citation attached.	No change detection on the source. The build loop only runs over what it is handed, and nobody hands it a change feed (Change Data Capture).	Drive ingestion from a change signal with a high-water mark, and reconcile document counts and versions against the source on a schedule (Reconciliation).
A parser or extractor is upgraded.	Answers about tables and headings get worse; nothing else changes and no job fails.	Extraction output changed shape while row counts stayed similar, so no volume check fired and no version was recorded (Semantic Changes).	Version the extractor on every chunk row, and monitor chunks-per-document by document rather than in aggregate (Volume Anomalies).
Chunk size is changed to fix one complaint.	Some questions improve, unrelated ones regress, and nobody can reproduce either.	The index now holds two grains simultaneously and the retriever cannot distinguish them (Chunking Pipelines).	Treat a chunking change as a full rebuild into a new index, evaluated before the switch — the same shape as any grain change to a fact table (Grain: What Does One Row Represent?).
The embedding model is upgraded.	Retrieval quality collapses for part of the corpus and is fine for the rest.	A partial re-embed left two vector spaces in one index, which is a half-completed data migration (Re-embedding).	Build the new index beside the old one, validate, then switch atomically; never write two model versions into one index (Atomic Publish).
Trace volume rises and the pipeline sheds load.	The incident you are investigating has no trace, while ordinary successful runs have plenty.	Head-based sampling or backpressure dropping, both of which discard uniformly and therefore discard the rare case you needed (Sampling Without Throwing Away the Evidence).	Sample by outcome rather than uniformly: keep every error, every low-confidence run and every case with negative feedback, and downsample the successes (Evaluation Data Pipelines).
A business policy changes.	The evaluation suite stays green while production answers get worse.	A curated set with no refresh path, whose expected answers encode a world that has moved (Golden Datasets).	Rebuild the set periodically from recent sampled traces, version it, and record which policy version each case was written against (Data Contracts).

How to build it

Most important first.

Give every one of them an owner, a written grain statement and a freshness target, published where a consumer can find them. That is the whole of the discipline and it takes an afternoon (Data Products, Dataset Documentation).
Make the corpus and the index derived, never authored: the only way a vector enters the index is by the pipeline running over a source document you still hold. A hand-inserted vector is the agentic equivalent of editing a fact table during an incident (Keeping Raw History: The Recovery Position and the Liability).
Record the version of everything that decides meaning — source document version, extractor version, chunker version, embedding model version, prompt version, tool schema version. Version columns are cheap and they are the only thing that makes a quality regression attributable (Vector Data Engineering).
Treat the trace archive as a data product with a classification and a retention policy on the day it is created, not after the first audit (Data Classification, Data Retention).
Build the evaluation set from production traces through an explicit sampling and privacy pipeline, so it can be rebuilt and so it drifts *with* reality rather than away from it (Evaluation Data Pipelines).
Monitor corpus freshness the way you would monitor a serving table. The age of the oldest un-reindexed source document is a single number and it belongs on a dashboard (Freshness Monitoring).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

The corpus guarantees, at best, that every source document present at the last successful pipeline run is represented by chunks in the index. It does not guarantee that every document currently in the source is represented, and the gap between those two sentences is the real freshness of retrieval (The Freshness SLO).
The index guarantees a nearest-neighbour answer within its own vector space. It does not guarantee relevance, it does not guarantee that a better chunk exists, and it has no way to report that the answer was never ingested at all (Vector Search: Embeddings, Similarity and ANN).
The trace stream is at-least-once and unordered with respect to real time: a span can arrive twice, and a later turn can be written before an earlier one (At-Least-Once Delivery).
The evaluation set guarantees only that its cases were true of the sample they came from at the moment they were sampled. Nothing about it is representative unless the sampling was designed to make it so.
Nothing anywhere in the stack guarantees that the answer given is supported by the chunk that was cited. That is a property you have to measure, and measuring it is the entire purpose of the evaluation pipeline (RAG Evaluation).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The highest-value check is a coverage reconciliation: count source documents in the system of record for a closed period, count distinct source documents represented in the index, and alert on the difference. It catches parse failures, silently skipped file types, and an ingestion job that has been exiting zero while processing nothing (Reconciliation).
It misses everything about quality *inside* a document. A PDF that extracted as a page of ligature garbage counts as covered, and so does one whose tables became a column of stray digits.
It also cannot see the document that was never in the source system to begin with — the most common cause of "the agent does not know that", and the one no pipeline check will ever find.

Freshness

Retrieval freshness is a per-document property, never a per-index one. An index refreshed hourly still serves a document that failed to parse three weeks ago, and an average across the corpus hides exactly that document (Freshness Monitoring).
The clock that matters starts when someone edited the policy page, not when the connector noticed. Measuring freshness from the pipeline run rather than from the source change reports a number that is always good and never true.
Trace freshness is a debugging property rather than a reporting one. An archive that lands hourly is fine for evaluation and useless during an incident, which is an argument for two paths over the same events rather than one compromise between them (Lambda Architecture).

When the schema or meaning changes

Schemas here change in the ordinary way: a metadata field added to chunks, a new column on traces. The ordinary compatibility rules apply and nothing about this domain is special (Schema Evolution).
What is unusual is how often the *meaning* changes with no schema change at all — a new chunking strategy, a different embedding model, an edited system prompt. Every one of those makes yesterday incomparable with today, and none appears in a schema diff (Semantic Changes).
The defence is the same as everywhere else in this domain. Version the thing that decides meaning, write the version on the row, and refuse to compare across versions without saying that is what you are doing (Data Contracts).

How to re-run this safely

The corpus and the index are rebuildable if — and only if — you kept the raw documents, the extraction output, and the versions of chunker and embedder that produced each row. With those, a bad ingest is a re-run; without them it is archaeology (Reprocessing vs Retrying).
Traces and feedback are not rebuildable. They are captured once from a production event that will not happen again, so losing a week of traces permanently loses a week of evaluation material. That puts trace durability in a different class from index durability, and it is usually the reverse of how the two are treated.
Rebuild beside, never in place. An index being rewritten while retrieval reads it serves half a corpus, and half a corpus retrieves with complete confidence (Atomic Publish).

What can go wrong

Failure modes

An ingestion job that succeeds while extracting nothing, because a parser returned an empty string rather than raising (The Pipeline Succeeded. The Data Is Wrong.).
An index holding two embedding model versions at once after a partially completed re-embed, so distances are being compared across incompatible spaces (Re-embedding).
A trace pipeline that drops under backpressure, so the traces missing from the archive are exactly the ones from the incident you are investigating (Sampling Without Throwing Away the Evidence).
An evaluation set that quietly contains cases whose expected answers were produced by the system being evaluated.
A privacy filter that redacts the message body and not the tool arguments, which is where the account numbers actually are.
The mitigation failing: a coverage check comparing the index against a manifest generated by the same broken job it was meant to audit.

Misreads

"The vector store is the source of truth." It is a derived index. The source of truth is the document system, and treating the index as authoritative is how a deletion request becomes unenforceable (Source of Truth).
"Retrieval quality is a model problem." Most retrieval failures seen in production are ingestion failures — the document was never parsed, the chunk boundary split the answer, a metadata filter excluded it. None of those is fixed by a different model (The LLM Data Pipeline).
"We can add evaluation later." Evaluation sets are built from traces, traces are captured rather than derived, and the traces from before you started collecting them do not exist. Later means a different and worse dataset.
"This is machine learning, so a data team should not own it." Every artefact here is a table, a file or a stream with a grain and a freshness requirement. The skills that keep a fact table honest are the skills that keep a corpus honest.

Privacy, retention and access

The trace archive is usually the most sensitive dataset in an agent platform and usually the least governed, because it was created by an observability tool rather than by a data pipeline (Data Classification, Security-Safe Logging).
A retrieval corpus inherits the access rules of every document in it, and a vector index has no natural way to express them. Filtering by permission at query time is a design decision that must be made before ingestion, not bolted on after (Data Access Control, RAG and Agent Memory Security).
Deletion has to reach the source document, the extraction output, the chunk table, the vector index, the trace archive and any evaluation case built from it. Six places, and a pipeline that typically knows about one (Deletion Requests, Data Minimization).

Operating it

How you see it in production

Age of the oldest un-reindexed source document, per corpus. One number, and the most useful one on the page (Freshness Monitoring).
Chunk and vector counts per source document version, so a document that went from forty chunks to two after a parser upgrade is visible before anyone notices bad answers (Volume Anomalies).
Retrieval hit distribution — which chunks are ever returned. A corpus where a large share of chunks have never once been retrieved is either badly chunked or full of material nobody asks about, and both are worth knowing (Data Discovery).
Trace ingestion lag and trace drop rate, reported separately. A drop rate that is quietly non-zero turns every downstream evaluation figure into an estimate of something nobody defined (The Log Bill and What It Is Buying).

What changes at 10x and 100x

At ten times the corpus, ingestion stops being a single job and becomes an incremental pipeline with a high-water mark, because a full rebuild no longer fits the window (Incremental Processing, The High-Water Mark).
At a hundred times, the index stops being one index. Tenant, language and document-class partitioning become correctness concerns rather than performance ones, because a retrieval that crosses a tenant boundary is a security incident rather than a slow query (Multi-Tenant Isolation).
Trace volume scales with usage rather than with content, so it overtakes the corpus early and then keeps going. The first storage surprise in most agent platforms is the trace archive (Cardinality: The Label That Took Down Monitoring).

What drives cost here

The drivers are the ordinary ones wearing new labels: bytes retained, where traces dominate because they grow with usage rather than with content; work repeated, because every re-embed reprocesses the whole corpus; and bytes scanned, because evaluation runs read the trace archive (What Actually Drives Data Platform Cost).
The one genuinely new driver is per-item external compute. Embedding and generation are billed per unit of work by a service you do not run, which turns reprocessing into a bounded, chargeable event rather than a fixed cluster running for longer. That changes the shape of a backfill decision more than anything else in this module.
Trace retention is the driver that grows without anyone deciding it should, because traces are written by production and deleted by nobody (Storage Lifecycle).

What this approach costs

Treating these as data products costs version columns, a scheduled pipeline, a catalog entry and a named owner — real work that produces nothing a demo can show. It buys the ability to answer "why did it say that" in minutes rather than never.
Full-fidelity traces are simultaneously the most useful debugging asset and the largest privacy liability the platform holds. Every retention and redaction choice moves along that single axis, and there is no setting that is good at both ends (Agent Observability Data).
Rebuildability requires keeping raw documents and extraction output as well as the index, which multiplies storage for a corpus you already stored once. That is worth it right up until it is not, and the honest test is whether you have ever actually needed to reprocess.

Agent dataset inventory

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

Agent dataset inventory

An agent given warehouse credentials inherits every access-control gap in the warehouse, and it will use all of them helpfully.

Datasets reachable

6 of 6sim

Restricted and reachable

3sim

Row-level access

anything the credential allows

Audit log reachable

yes

Sensitivity	Agent access	Freshness
internal	read	Rebuilt nightly; today is incomplete until tomorrow.
internal	read	Incremental, roughly hourly.
restricted	summary-only	Continuous from change capture.
restricted	REACHABLE — should not be	Continuous.
restricted	REACHABLE — should not be	Continuous.
internal	summary-only	Daily.

	stg_customers
Contains	Names, email addresses, addresses.
Why that access	An agent that can select rows here can exfiltrate them into a summary, a log line, or a support reply. Aggregates are safe; rows are not.

With unscoped credentials the agent can read raw payloads nobody has enumerated and the audit log of who queried what. Neither is a hypothetical: an agent asked a broad question will find the widest table that seems relevant, and a raw events table always seems relevant.

ORG-SPECIFICThe right scoping depends on what the agent is for and on obligations that differ by jurisdiction and industry. The general rule holds: grant an agent the smallest set of datasets that answers its questions, and treat "the agent summarised it" as a read, because it is.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe six datasets, their grains and their failure classes are properties of retrieval-and-tool architectures generally, independent of which model, vector store or framework is used. What differs between stacks is which system holds each dataset, not what each one is able to lose.
SCALE-SPECIFICBelow a few thousand documents and a single team, rebuilding the whole index on every change is simpler and more correct than an incremental pipeline, and the versioning discipline can live in a README. The advice inverts once a rebuild stops fitting in a maintenance window or a second team starts adding documents.
ORG-SPECIFICOwnership, classification and catalog entries solve coordination problems that only appear when the team writing the documents is not the team running the agent. In a single-team product they are ceremony; across a content team, a platform team and a compliance function they are the only thing that stops the corpus becoming unattributable.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasevector-search

Backendagent-backends agent-audit-logs at-least-once-delivery

Securityrag-and-memory-security data-classification multi-tenant-isolation

Observabilityagent-observability agent-cost trace-sampling cardinality

Cloudai-workload-infrastructure object-storage

Agenticwhat-agentic-engineers-build the-agent-loop rag-overview agent-plus-rag tracing-agents golden-datasets

Domains that do not exist yet

— Distributed Systems owns the delivery and ordering semantics the trace stream inherits — why a span can arrive twice, why a later turn can be written before an earlier one, and what a replay of that stream actually replays.
— DevOps / Production Engineering owns how a prompt, a tool schema and a chunker version are versioned, deployed and rolled back. Every one of them changes the meaning of a dataset, so each deserves the review discipline of a schema migration rather than that of a config tweak.