Data Engineering for AI & Agents

Retrieval corpora, embeddings, evaluation sets and agent traces are data products with schemas, freshness, lineage and cost. Re-embedding is a data migration.

Data Engineering for Agents
▶ lab

An agent system is a data platform with a model in the middle: six datasets, each with a grain, an owner, a freshness target, and its own quiet way of rotting.

Q · An agent gave a customer the wrong answer on Tuesday. Which datasets did it read, which did it write, and who owns each one?
The LLM Data Pipeline
▶ lab

Documents to ingest to clean to chunk to metadata to embed to index to retrieval. Nine stages, nine promises, and most retrieval failures happen in the first three.

Q · A PDF lands in a shared drive. What has to happen, in what order, before an agent can retrieve one sentence from it — and what does each of those steps actually promise?
Chunking Pipelines
▶ lab

Chunking is not preprocessing and not a hyperparameter. It is the grain declaration for the retrieval index, and a boundary in the wrong place is the same class of error as a wrong fact-table grain.

Q · What does one row of your retrieval index represent — and which questions does that choice quietly make unanswerable?
Embedding Pipelines
▶ lab

Turning a corpus into vectors is a batch job with a metered external call in the middle. Keyed sink, watermarked input, work queue derived by difference — or it will not finish.

Q · Ten million chunks need vectors. How do you run that as a restartable, idempotent, cost-bounded batch job rather than a loop that dies at sixty per cent?
Re-embedding
▶ lab

A new embedding model makes every vector in the corpus stale. Recompute beside the old index and switch atomically — this is a data migration, and it obeys backfill rules exactly.

Q · The embedding model changes. What has to happen to a corpus of vectors, and why is that the same operation as any other backfill?
Vector Data Engineering
▶ lab

A vector is a row in a derived dataset. Source version, chunk strategy, embedding version, text hash and reindex status are the columns that make a corpus debuggable, rebuildable and governable.

Q · What has to sit beside a vector for a corpus to be explainable a year later — and which questions become permanently unanswerable for each column you did not write?
Evaluation Data Pipelines
▶ lab

Production traces, sampled, privacy-filtered and versioned into an evaluation dataset. The privacy filter is the step most often skipped, and the version is what makes a score comparable across runs.

Q · Where does an evaluation set come from, and what makes this month's score comparable with last month's?
Agent Observability Data
▶ lab

Prompt, model, tool calls, latency, tokens, outcome and feedback — as a high-cardinality event table with a classification and a retention policy, not as logs in a bucket.

Q · What has to be recorded about an agent run for it to be explainable a week later, and how much of what makes it explainable is the customer's own words?
Feature Pipelines
▶ lab

Raw events to transformations to features to two consumers. The characteristic failure is one logical column computed by two pipelines, and it is a data-engineering failure with a data-engineering fix.

Q · The same feature is computed by a batch job for training and by a service at request time. What makes those two numbers differ, and how would you find out?