Vector Data Engineering
A vector is a row in a derived dataset. Source version, chunk strategy, embedding version, text hash and reindex status are the columns that make a corpus debuggable, rebuildable and governable.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What has to sit beside a vector for a corpus to be explainable a year later — and which questions become permanently unanswerable for each column you did not write?
The retrieval path, which can only filter on metadata that was written at build time; the on-call engineer asking why a specific chunk came back; the migration job that needs to know which chunks are stale; the deletion process that must find every vector derived from one document; and the evaluation, which cannot attribute a regression to a change nobody recorded (Metadata Filtering).
One row is one vector for one chunk under one embedding model version, and the natural key is that pair. The vector store's index is a *projection* of that table for serving, not the table itself — a distinction that decides whether a corpus can be rebuilt, audited and migrated, or only queried (Embedding Pipelines).
Write vectors into the vector store with the chunk text and a document id attached, and treat the store as the database. It is one system instead of two, the store is genuinely good at what it does, and for a single-tenant corpus with one chunking strategy and no compliance obligations this is a reasonable place to be.
"Which documents are indexed at the current model version?" has no answer, because nothing records which version each vector was written under. The only way to find out is to re-embed everything and compare (Re-embedding).
- "Which documents are indexed at the current model version?" has no answer, because nothing records which version each vector was written under. The only way to find out is to re-embed everything and compare (Re-embedding).
- A document is deleted at the source. Its vectors are identifiable only if a document id was written onto every chunk — and if chunk ids were positional, the mapping is already wrong after the document was edited (Deletion Requests).
- A parser upgrade needs to be applied to PDFs only. Nothing records which extractor produced which chunk, so the choice is to re-run the whole corpus or to run nothing (Reprocessing vs Retrying).
- Retrieval quality drops. Two changes shipped that week and neither is recorded on the rows, so the regression cannot be attributed and the investigation ends in a guess (Semantic Changes).
- Permission-aware retrieval is requested. The permission fields were never written, and they cannot be added at query time — adding them means rebuilding every chunk in the corpus (RAG and Agent Memory Security).
- The index is corrupted or the store is migrated. The vectors existed only inside it, so rebuilding means paying the entire embedding cost again for data that was never lost, only unreachable (Backup Strategy).
What is actually happening
- The vector store is a serving index, and treating it as the system of record is the same mistake as treating a search index as the source of truth for orders. The system of record is a chunk table and a vector table you own; the index is a projection you can drop and rebuild (Source of Truth).
- Metadata is written at build time and only at build time. A filter you might want at query time is a column you had to write months earlier, which makes metadata design a schema decision with the same one-way character as a fact table's grain (Metadata Filtering, Grain: What Does One Row Represent?).
- Five separate versions decide what a vector means: source document version, extractor version, chunk strategy version, embedding model version, and the client library that tokenised the input. Each can change independently, none changes a schema, and all of them change meaning (Semantic Changes).
- Mixed-version vectors in one index are not comparable and nothing raises an error. A vector is a list of numbers; the store ranks them faithfully and returns a confident ordering computed across two unrelated spaces (Re-embedding).
- "Is the corpus up to date" is only answerable if reindex state is explicit per chunk and version — pending, embedded, indexed, dead-lettered, superseded. Without that column the question degrades into a count comparison that cannot distinguish "not started" from "given up on" (The High-Water Mark).
- The metadata beside a vector plays the role a dimension table plays beside a fact: it is what turns a similarity search into a query with predicates, and its cardinality decides whether a filter is selective or decorative (Dimension Tables, Cardinality: The Label That Took Down Monitoring). Filtering interacts with approximate search in a way that surprises people. A filter applied after the nearest-neighbour search removes results from a fixed candidate set and can return fewer matches than asked for; a filter applied before or during the search constrains the traversal itself. Which one a store does is a correctness-shaped property, not a performance detail (Vector Search: Embeddings, Similarity and ANN).
The table is the system of record; the index is a projection
Everything in this lesson follows from one decision: whether the vector store holds your data or serves it. If it holds it, every question about history, versions and completeness has to be asked of a system built to answer a different question. If it serves it, the store becomes replaceable and the interesting queries become ordinary SQL (Source of Truth).
The schema below is deliberately boring. It is a chunk table, a vector table keyed by chunk and model version, and a handful of columns that exist purely so that questions can be asked later. Nothing about it is specific to retrieval — it is the same metadata discipline this domain applies to any derived dataset, which is exactly the point (Metadata: Technical, Operational and Business).
The two queries underneath are the ones that are impossible without the columns. "Which documents are stale at the current version" and "which rows did the thing that changed produce" are the first and second questions of every corpus investigation, and both are one join away when the versions are on the rows and unanswerable when they are not.
1-- One row per retrievable unit. Owned by you, not by the vector store.2CREATE TABLE chunk (3 chunk_id TEXT PRIMARY KEY, -- f(doc_id, doc_version, strategy, offsets)4 doc_id TEXT NOT NULL, -- deletion and citation both need this5 doc_version TEXT NOT NULL, -- which revision of the source document6 extractor_version TEXT NOT NULL, -- which parser produced the text7 chunk_strategy TEXT NOT NULL, -- which boundary rules produced this unit8 byte_start INT NOT NULL, -- makes a citation checkable9 byte_end INT NOT NULL,10 text TEXT NOT NULL,11 text_sha256 TEXT NOT NULL,12 doc_class TEXT NOT NULL, -- filter, and re-run selector13 tenant_id TEXT NOT NULL,14 classification TEXT NOT NULL, -- drives masking and retention15 read_scope TEXT NOT NULL, -- captured from the source, never inferred16 effective_from DATE, -- policies and prices have dates17 effective_to DATE18);19 20-- One row per chunk per embedding model version. The composite key is what21-- lets two versions coexist in the TABLE while never coexisting in one INDEX.22CREATE TABLE chunk_vector (23 chunk_id TEXT NOT NULL,24 embedding_model_version TEXT NOT NULL, -- model + revision + client library25 chunk_text_sha256 TEXT NOT NULL, -- what was actually embedded26 reindex_state TEXT NOT NULL, -- pending|embedded|indexed|dead|superseded27 index_build_id TEXT, -- which projection this row reached28 embedded_at TIMESTAMP NOT NULL,29 vector VECTOR,30 PRIMARY KEY (chunk_id, embedding_model_version)31);32 33-- Question 1: what is stale right now, and by how much?34SELECT c.doc_id, c.doc_version, COUNT(*) AS chunks_missing_current_vector35FROM chunk c36LEFT JOIN chunk_vector v ON v.chunk_id = c.chunk_id37 AND v.embedding_model_version = :current_version38 AND v.reindex_state = 'indexed'39WHERE v.chunk_id IS NULL40GROUP BY c.doc_id, c.doc_version41ORDER BY chunks_missing_current_vector DESC;42 43-- Question 2: the parser for PDFs was wrong. What exactly has to be re-run?44SELECT chunk_id45FROM chunk46WHERE extractor_version = 'pdf-v3'47 AND doc_class = 'contract';Neither query touches a vector. Both are the questions asked in every corpus incident, and both are answerable only because the versions were written on the rows at build time — which is a decision made months before the incident by someone who had no specific reason to expect it.
Every hop can corrupt something, and only one of them errors
Walking the chain from document to answer makes the metadata argument concrete. Each node holds something, and each can corrupt what it holds in a way the next node cannot detect — because every hop passes on a value that looks structurally correct regardless of whether it is right.
Read the couldCorrupt column and count the entries that raise an exception anywhere: one, and it is the dimension mismatch, which is the luckiest failure in the whole chain precisely because the write is rejected. Everything else propagates quietly into an answer a human reads (The Pipeline Succeeded. The Data Is Wrong.).
This is also the map for a debugging session. When retrieval returns something inexplicable, the question is which node last touched that chunk, and the version columns on the row are what let you answer it without re-deriving the chain by hand (Data Lineage, Lineage Debugging).
- Source document at a version
holds The authored content, with its own permissions, effective dates and classification.
could corrupt Being edited or withdrawn without a change signal, so everything derived from it describes a version that no longer exists (Change Data Capture).
↑ reads from - Extraction output (extractor version)
holds Text plus structure, reconstructed from the file by a parser.
could corrupt A parser upgrade that changes reading order or flattens tables — same row counts, different meaning, no error (Semantic Changes).
↑ reads from - Chunk row (chunk strategy version, offsets)
holds One retrievable unit, addressable back into a stored document version.
could corrupt A boundary through a table or a procedure, or positional ids that renumber when a paragraph is inserted (Chunking Pipelines).
↑ reads from - Metadata on the chunk
holds Tenant, class, permissions, effective dates, classification — captured at build time.
could corrupt Values that were true when written and are not now, or permissions inferred from a folder path rather than captured from the source (Data Access Control).
↑ reads from - Vector row (embedding model version, text hash)
holds Coordinates in one model version's space, plus what text produced them.
could corrupt A vector computed from chunk text that has since changed, or a version label that does not match the client library that produced it (Re-embedding).
↑ reads from - Serving index (build id)
holds A projection of the vector rows, organised for approximate nearest-neighbour search.
could corrupt Holding two model versions at once, or silently missing rows the table has — a copy that drifts and cannot report it (Reconciliation).
↑ reads from - Retrieval result
holds k chunks, ranked, with metadata filters applied.
could corrupt A post-search filter leaving fewer than k results, a query encoded by a different version, or a chunk whose permissions changed after it was indexed (RAG and Agent Memory Security).
Seven hops, and exactly one — a dimension mismatch at the index write — fails loudly. Every other corruption produces a well-formed, confidently ranked result, which is why the version columns are not documentation but the only available instrumentation.
Adding a field later is a rebuild, not an ALTER
The most expensive sentence in this module is "we can add that metadata later". Metadata is written per row at build time, so a new filter is only usable once every row carries it — and rows are made by a pipeline whose expensive stage is metered per item (Embedding Pipelines).
The diff below is a real and ordinary request: retrieval should respect document-level permissions and effective dates. In schema terms it is purely additive and every compatibility checker will wave it through. In corpus terms it means every chunk written before today lacks the fields, so any filter using them either excludes the entire historical corpus or silently ignores it (Nullability & Defaults).
Read the silent column. The consumer that breaks loudly is the easy one. The dangerous rows are the ones where a filter quietly matches a subset — retrieval still returns ten results, the agent still answers, and the only symptom is that the right document was never a candidate (Missing Rows).
- chunk_id
- doc_id
- doc_version
- chunk_strategy
- text
- text_sha256
- doc_class
- chunk_id
- doc_id
- doc_version
- chunk_strategy
- text
- text_sha256
- doc_class
- tenant_id
- read_scope
- effective_from
- effective_to
change Four metadata fields added to the chunk row so retrieval can filter by tenant, by reader permission and by the date a policy was in force. Additive in the schema; a full corpus rebuild in practice, because rows written before the change carry none of them.
| Consumer | Effect | How it shows up |
|---|---|---|
| Retrieval filtering on `read_scope` | Every chunk written before the change is excluded, because the field is null. Recall collapses for the historical corpus and the query still returns a full page from whatever remains. | Silently — no error, wrong result |
| Retrieval filtering on `effective_from`/`effective_to` | Superseded policy text is indistinguishable from current text for old rows, so the agent answers from a rule that expired last year and cites it correctly (Slowly Changing Dimensions). | Silently — no error, wrong result |
| Tenant isolation via `tenant_id` | Old rows have no tenant, so either they are visible to everyone or to no one, depending on how the filter treats null — and both outcomes are decided by an operator precedence nobody reviewed (Multi-Tenant Isolation). | Silently — no error, wrong result |
| The embedding job | Nothing. Vectors are a function of text, and none of these fields is text, so a metadata-only backfill needs no re-embedding — the one piece of good news in the diff (Embedding Pipelines). | Loudly — it raises |
| The evaluation harness | Recall figures computed before and after the change are not comparable, because the candidate set is now filtered. Plotting them on one axis reports a regression that is a definitional change (RAG Evaluation). | Silently — no error, wrong result |
| The deletion process | Improves: tenant_id and doc_id make a scoped deletion expressible. Before the change it was a full scan of chunk text (Deletion Requests). | Loudly — it raises |
When the filter meets the index
Metadata filtering looks like a WHERE clause and behaves like one only if the index applies it during the search. Many approximate indexes search first and filter afterwards, which means the filter is applied to a fixed candidate set — and a selective filter over a small candidate set can leave you with almost nothing, while reporting success (Vector Search: Embeddings, Similarity and ANN).
This is the one place in the module where an implementation detail of the store changes what your data means. The same corpus, the same filter and the same query can return a full page of results in one store and two results in another, and neither is malfunctioning (Index Types: B-tree, Hash, Partial, Expression, Covering, Full-Text).
The engineering response is not to pick a store on this basis alone. It is to know which behaviour yours has, to test it under your most selective filter, and to design the corpus so that the most selective dimensions — tenant above all — are index boundaries rather than predicates (Multi-Tenant Isolation).
All tenants, all document classes and all languages share one index. Every query carries a tenant predicate. Correctness depends on that predicate being present and correctly applied on every retrieval path, including the ones added later by someone who copied an older call site.
One index per tenant — or per tenant group — with document class and language as ordinary predicates inside it. The retrieval path selects an index from the authenticated identity before any search happens, and a missing predicate can no longer cross a tenant boundary.
A metadata filter is applied by a ranking system as part of a search whose recall is already approximate; an index boundary is applied by a routing decision before the search exists. That difference converts a class of correctness failure — one caller forgetting a predicate, one filter applied post-search over too few candidates — into something structurally impossible. It costs duplicated infrastructure and it is the same argument that puts tenant separation at the schema level rather than in every query in a multi-tenant database (Tenant Isolation).
Whether metadata predicates constrain the traversal or are applied after it, whether a collection can hold more than one vector dimension, how many metadata fields can be indexed for filtering, and how selective filters interact with recall are properties of the specific store and version, and they change between releases. Test your own most selective filter against your own corpus rather than reading the behaviour off a comparison table.
How to build it
Most important first.
- Own the chunk table and the vector table outside the store, and treat the index as a rebuildable projection of them. That one decision makes migration, audit, deletion and store replacement into ordinary operations (Source of Truth).
- Make chunk ids deterministic — a function of document id, document version, chunk strategy version and byte offsets — so a re-run is an upsert rather than an append, and so every vector points back at a real span of a real document (Idempotent Data Pipelines, Chunking Pipelines).
- Write the five versions on every row, plus the chunk text hash. Version columns cost bytes and they are the only thing that makes a regression attributable or a targeted re-run possible (Data Lineage).
- Key the vector table on
(chunk_id, embedding_model_version)so a model change inserts rather than overwrites, and so two versions can coexist in the *table* while never coexisting in one *index* (Upserts and Merges). Record reindex state explicitly and treat it as a state machine, so "remaining work" is a query that reaches zero rather than a count that stalls forever on chunks the model will never accept (Embedding Pipelines). - Write permissions, tenant, effective date, document class and classification at chunk time from the source system's own model — never inferred later from the text, because an access rule that is right most of the time is the worst kind (Data Access Control, Data Classification). Separate tenants into separate indexes rather than relying on a metadata filter alone. A filter is a query-time promise; an index boundary is a structural one, and a retrieval that crosses tenants is a security incident rather than a bad result (Multi-Tenant Isolation).
- Reconcile the index against the vector table on a schedule. The index is a copy, copies drift, and nothing in a vector store will tell you that it is missing rows (Reconciliation).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The vector table guarantees that every row is attributable: which document, which version, which chunk strategy, which model, and what text was actually embedded. That is a much stronger property than anything the index provides and it is entirely a schema decision (Citations).
- The index guarantees the nearest neighbours of a query vector among the vectors it currently holds, under the filters supplied. It does not guarantee it holds everything the table holds, and only a reconciliation will tell you (Vector Search: Embeddings, Similarity and ANN).
- Nothing guarantees that two vectors in one index came from the same model. That invariant has to be asserted by a check you write, because no store enforces it (Data Tests).
- Metadata filters guarantee only what was true at build time. A document whose permissions changed yesterday is filtered by yesterday's permissions until it is rebuilt, which is an access-control property people consistently assume they have and do not (RAG and Agent Memory Security).
- Approximate indexes do not guarantee that the true nearest neighbour is returned at all. That is the trade being made in exchange for tractable search, and it is worth stating because it interacts with filtering in ways that look like data loss (Index Types: B-tree, Hash, Partial, Expression, Covering, Full-Text).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that pays for itself immediately is version uniformity: count distinct embedding model versions, chunk strategy versions and extractor versions present in the serving index. Anything other than one, per column, is either a migration in progress or a bug (Data Tests).
- It misses versions that are labelled identically and produced differently — a client library upgrade that changed tokenisation while the recorded model string stayed the same. Only a re-embed of a fixed sample compared for equality catches that (Re-embedding).
- Pair it with an index-versus-table reconciliation: rows in the vector table at the current version, compared with rows in the index, compared with distinct chunks in the chunk table. Three counts, and the differences between them localise a fault to the embed stage, the index write, or the chunking stage (Reconciliation).
- Corpus freshness is a per-document, per-version property, and the useful statistic is a maximum rather than a mean: the oldest chunk not yet reindexed at the current version is the one that will embarrass you, and it vanishes into an average (Freshness Monitoring).
- There are two clocks and they are frequently confused. One measures how long ago a source document changed; the other measures how long ago the index was written. A pipeline that runs hourly over a source it re-scans nightly is nightly, and only the first clock tells you that (The Freshness SLO).
- Metadata staleness is its own freshness problem with no retrieval symptom. Permissions, effective dates and document classes written at build time drift from the source silently, and the index keeps filtering on values that were true last quarter (Semantic Changes).
- Adding a metadata column is backward compatible in the schema and useless in practice until the whole corpus carries it, because filters must be applied to every candidate. A new filter is therefore usually a rebuild rather than an
ALTER(Schema Evolution). - Changing what an existing metadata field means — a document class taxonomy that gains a category, a permission model that changes granularity — is a silent breaking change to every filter written against it (Semantic Changes, Breaking Schema Changes).
- Version columns are the mechanism that makes any of this survivable. With them, "which rows were produced by the thing that changed" is a query; without them it is the whole corpus by default (Impact Analysis).
- The index schema and the metadata contract should be versioned together and published, because the retrieval path is a second consumer that must agree with the build path about every field name and every value domain (Data Contracts).
- If the vector table is owned outside the store, losing the index costs a rebuild from rows you still have — no embedding spend, no source access, no parsing. If the index was the only copy, the same event costs the entire corpus' worth of external compute again (Backup Strategy).
- Targeted repair is what the version columns buy. "Re-run everything produced by extractor v3 for PDFs" is a
WHEREclause; the same request without version columns is a full corpus rebuild with a budget conversation attached (Planning a Backfill). - Rebuild into a new index and switch, never in place, so a failed rebuild costs storage rather than availability and the previous index stays available as a rollback target (Atomic Publish, Rolling Back Data).
What can go wrong
- The store used as the system of record, so an index loss or a store migration costs the full embedding spend again.
- Positional chunk ids, so editing one paragraph renumbers a document and orphans every downstream reference to it (Chunking Pipelines).
- Mixed versions in one index producing confidently ranked nonsense with no error anywhere (Re-embedding).
- Permissions inferred from folder paths or chunk text rather than captured from the source, producing an access rule that is right most of the time (Data Access Control).
- Metadata written once and never refreshed, so retrieval filters on a permission model that changed last quarter.
- The mitigation failing: a reconciliation that counts rows in the index without filtering by model version, and therefore reports a complete corpus throughout a partial migration.
- "The vector database is the database." It is an index. The system of record is the document store and the tables you derive from it, and treating the index as authoritative is how a deletion request becomes unenforceable (Source of Truth).
- "Metadata can be added later." Metadata is written per row at build time. Adding a field means rebuilding every row that lacks it, which for a corpus of any size is a migration rather than a change (Re-embedding).
- "Same dimensions means compatible vectors." Dimensionality is a shape, not a space. This one is dangerous because the store accepts the write and the query returns results (Vector Search: Embeddings, Similarity and ANN).
- "A metadata filter is an access control." It is a query-time predicate over values captured at build time, applied by a system whose job is ranking. Real isolation is a separate index and an authorisation check outside the retrieval path (RAG and Agent Memory Security).
- "Retrieval returned nothing, so the answer is not in the corpus." It may be a filter excluding everything, an approximate index missing the neighbourhood, or a version mismatch. Absence of results is not evidence of absence of content (Missing Rows).
- Classification, permissions and retention have to be columns on the chunk row, because governance can only act on what it can select. A corpus without them is governable only by deleting all of it (Data Classification, Data Retention).
- Deletion must reach raw, extraction output, chunk table, vector table at every model version, and every index that projects them. Deterministic ids linking those five are what turn a deletion request into an operation rather than a search (Deletion Requests).
- Access control that lives only as a retrieval filter fails open under any bug in the retrieval path. Separate indexes per tenant and an authorisation check outside the search make the failure mode a missing result rather than a leak (Multi-Tenant Isolation, Least Privilege).
Operating it
- Distinct values of each version column present in the serving index — model, chunk strategy, extractor. Each should be one, and this is the cheapest standing monitor in the module (Data Tests).
- Chunks by reindex state, so "pending" and "dead-lettered" are separate numbers rather than a single stalled percentage (Pipeline Metrics).
- Age of the oldest chunk not yet at the current version, per corpus and per document class (Freshness Monitoring).
- Index row count versus vector table row count at the current version, reconciled on a schedule, because a serving copy drifts and says nothing about it (Reconciliation).
- Filter selectivity in production: how many candidates survive each metadata predicate. A filter that never excludes anything is metadata you are paying to store and not using, and one that excludes almost everything is probably a bug (Cardinality: The Label That Took Down Monitoring).
- At ten times the corpus, metadata design stops being optional because full rebuilds no longer fit in a window and every repair must be targeted (Incremental Processing).
- At a hundred times, one index becomes several — by tenant, language or document class — and the router that picks between them becomes part of the retrieval contract rather than an implementation detail (Data Marts).
- High-cardinality metadata (per-user permissions, per-document ACLs) is where filtering gets genuinely hard, because a highly selective filter over an approximate index can leave the traversal with almost nothing to walk (Partition Cardinality).
- Tenant count scales isolation risk faster than data volume scales anything. Separate indexes per tenant cost more and convert a class of correctness bug into a structural impossibility (Multi-Tenant Isolation).
- Vector storage is dense, fixed-width per row and linear in chunk count times dimension — one of the few costs in this domain computable in advance rather than observed afterwards (What Actually Drives Data Platform Cost).
- Metadata is cheap to store and expensive to add later, because adding it means recomputing rows you already paid for. The asymmetry is the entire argument for writing columns you are not yet sure you need (Compute Waste).
- Holding two model versions in the vector table doubles that storage for the overlap period, which is the price of a migration that can be rolled back (Re-embedding).
- Index memory and query cost scale with vector count and dimension, so chunk size is simultaneously an embedding cost lever, a storage lever and a serving cost lever (Chunking Pipelines).
- Owning the chunk and vector tables outside the store means two systems, two schemas and a reconciliation. It buys rebuildability, targeted repair, auditability and the freedom to replace the store — which is the difference between a corpus you operate and one you host.
- Writing every version column and hash costs storage on every row and is impossible to add retroactively. The trade is paying a small permanent cost against the option value of every future targeted re-run.
- Separate indexes per tenant cost duplicated infrastructure and remove an entire class of cross-tenant retrieval bug. Above a small tenant count that is nearly always the right trade (Tenant Isolation).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat a vector is a row in a derived dataset, that metadata must be written at build time, and that mixed versions are incomparable hold for every store and every model. What differs is how much of the chunk table a given store can hold beside its vectors, which changes how much duplication you accept, not whether the table has to exist.
- ENGINE-SPECIFICWhether a metadata predicate constrains the nearest-neighbour traversal itself or is applied to the results of an unconstrained search differs by index implementation, and it decides whether a highly selective filter returns fewer than the requested number of matches or simply takes longer. Test the behaviour of your own store under a selective filter rather than assuming.
- SCALE-SPECIFICFor a single-tenant corpus of a few thousand chunks with one chunking strategy, the store alone is genuinely sufficient and an external chunk table is bookkeeping. The advice inverts at the point where a rebuild no longer fits in a window, a second tenant appears, or a deletion request must be provably complete.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what it means for a serving index to be a copy that can silently diverge from the table it projects, and why reconciliation rather than trust is the only workable relationship between the two.
- — DevOps / Production Engineering owns the rollout of a metadata schema change whose blast radius is every row written after it, and the canary corpus that should precede it.