Embedding Pipelines
Turning a corpus into vectors is a batch job with a metered external call in the middle. Keyed sink, watermarked input, work queue derived by difference — or it will not finish.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Ten million chunks need vectors. How do you run that as a restartable, idempotent, cost-bounded batch job rather than a loop that dies at sixty per cent?
The vector index, which needs one vector per chunk at one model version and has no way to tell you it is short; the retrieval path, whose recall degrades silently in proportion to how much of the corpus is unembedded; and whoever approves the spend, who needs the run to be bounded before it starts rather than measured after it ends (What One Agent Run Costs, and Which Term Dominates).
One unit of work is one chunk at one embedding model version. Not one document, and not one batch request — the batch is a transport detail. That is why the natural key of the vector table is the pair (chunk_id, embedding_model_version), and why anything keyed on chunk_id alone cannot survive a model upgrade without destroying the vectors it is replacing (Surrogate Keys).
A loop over the chunk table that calls the embedding service once per chunk and inserts what comes back. It reads clearly, it debugs easily on a hundred rows, and on a corpus that finishes inside a lunch break it is the correct answer — a queue, a pool and a watermark would all be overhead.
The run reaches sixty per cent and the process is killed. Nothing durable recorded which chunks were done, so the options are to start from zero and pay for the first sixty per cent a second time, or write a recovery script under pressure (Checkpointing).
- The run reaches sixty per cent and the process is killed. Nothing durable recorded which chunks were done, so the options are to start from zero and pay for the first sixty per cent a second time, or write a recovery script under pressure (Checkpointing).
- A timed-out request is retried and the original had actually succeeded. Two vectors now exist for one chunk, and retrieval returns both as if they were independent evidence (Duplicate Rows).
- The loop is parallelised to go faster, immediately meets the service's rate limit, and every worker retries at once — so aggregate throughput falls below what the serial version achieved (Retry Storms: The Load You Generated Yourself).
- One chunk exceeds the accepted input length. The call fails, the exception is caught and logged, the run continues, and the corpus is now missing precisely its longest chunks — which are disproportionately the ones containing tables.
- The job succeeds while the chunk table is still gaining rows. The index is complete with respect to a snapshot that stopped existing during the run, and the exit code says nothing about it (The High-Water Mark).
- The cost is discovered afterwards, because nothing counted the units of work before the run started and the loop had no cap.
What is actually happening
- This is a batch pipeline whose expensive stage is a network call to a metered external service, a shape most data pipelines do not have. The usual assumptions all fail: compute is not fungible, retrying is not free, and the bottleneck is not your cluster (Calling Something You Do Not Control).
- The unit of transport is a batch holding many chunks; the unit of work is one chunk. Conflating them produces the classic partial-batch failure, where one unacceptable item fails a request containing hundreds of good ones and a naive handler discards all of them.
- Throughput is bounded by the service rather than by your workers. Past the point where the limiter engages, added parallelism converts capacity into retries and retries into a queue that grows faster than it drains (Backpressure).
- The work is embarrassingly parallel and perfectly idempotent if the sink is keyed. Both properties come from the data model rather than from the job, which is why the schema decision has to precede the concurrency decision (Idempotent Data Pipelines).
- For pipeline purposes a vector is a deterministic function of chunk text and model version, so a completed unit never needs recomputing. The entire job design follows from that one fact: record what is done, never redo it, and make the record survive a crash (The High-Water Mark).
- The output is dense and fixed-width per row, so vector storage grows linearly and predictably with chunk count. It is one of the very few costs in this domain you can compute in advance instead of observing after the fact (What Actually Drives Data Platform Cost).
The work queue is a difference, not a cursor
Almost every problem with these jobs comes from trying to remember progress. A cursor in a file, an offset in memory, a "last processed id" in a config table — each of them is a second source of truth that can disagree with the data, and each of them is wrong after exactly one crash.
The alternative is to store no progress at all and derive the remaining work by asking the data. Which chunks have no vector at the target version, and are not already in the dead-letter table? That query is the queue. It is correct after a crash, correct after a partial run, correct when two runs overlap, and correct when someone adds chunks in the middle — and it requires nothing to be remembered between runs (Idempotent Data Pipelines).
The watermark is the other half. Without it, "embed everything" is evaluated against a set that grows while the job reads it, and the job can never truthfully report completion. With it, the run has a defined input, a defined output and a defined remainder, which is the minimum needed to say anything honest about coverage (The High-Water Mark).
1-- One row per chunk per model version. The composite key is what makes a2-- retry a no-op and an upgrade an INSERT rather than a destructive UPDATE.3CREATE TABLE chunk_vector (4 chunk_id TEXT NOT NULL,5 embedding_model_version TEXT NOT NULL,6 chunk_text_sha256 TEXT NOT NULL, -- what was actually embedded7 embedded_at TIMESTAMP NOT NULL,8 vector VECTOR, -- type varies by store9 PRIMARY KEY (chunk_id, embedding_model_version)10);11 12-- The work queue for one run. Derived, never stored: a restart re-derives it,13-- so there is no cursor to go stale and no progress file to disagree with the14-- data. The watermark gives the run a fixed input set.15SELECT c.chunk_id, c.text16FROM chunk c17LEFT JOIN chunk_vector v ON v.chunk_id = c.chunk_id18 AND v.embedding_model_version = :target_version19LEFT JOIN embed_dead_letter d ON d.chunk_id = c.chunk_id20 AND d.embedding_model_version = :target_version21WHERE c.created_at <= :watermark -- a defined input set, not a moving one22 AND v.chunk_id IS NULL -- not embedded at this version yet23 AND d.chunk_id IS NULL -- and not already given up on24ORDER BY c.chunk_id; -- stable order, so partitions are stable25 26-- Completeness is then one query rather than a claim about the exit code:27-- chunks at watermark = vectors at version + dead-lettered at versionThe interesting line is the second LEFT JOIN. Without a dead-letter table, a chunk the model will never accept is selected by every run forever, so the job never converges and the operator learns to ignore its error count. Recording the give-up decision is what makes "remaining work" a number that reaches zero.
How these runs actually fail
Every row below is a failure of the job rather than of the model, and every one of them produces either a silent gap in the corpus or a run that costs twice what it should. None of them is exotic; the first three appear in nearly every first implementation.
The pattern to notice in the cause column is that four of the six are the same mistake wearing different clothes: treating the batch, the process or the clock as the unit of work instead of the chunk. Once the unit of work is a row with a key, retries, restarts and partial failures all stop being special cases.
The response column is where the external dependency shows through. Backing off, capping concurrency and isolating budgets are not techniques this domain usually needs — a Spark job does not need to be polite to its own cluster — and they are the part of this pipeline that is genuinely borrowed from service engineering rather than from data engineering (Tool Errors, Retries and Timeouts).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The service returns rate-limit responses under load. | Aggregate throughput falls below what one worker achieved, while error counts climb with every worker added. | Each worker retries immediately on rejection, so the limiter sees more load exactly when it is shedding (Retry Storms: The Load You Generated Yourself). | Cap concurrency below the limit, back off with jitter, and treat sustained limiting as a signal to remove workers rather than add them (Without Jitter, Every Client That Failed Together Retries Together). |
| One item in a batch exceeds the accepted input length. | A batch of hundreds fails, a handler logs and continues, and the corpus ends up missing its longest chunks. | The batch is being treated as the unit of work when the unit of work is a single chunk. | Split the failed batch, retry per item, dead-letter what still fails with a reason, and add a pre-flight length check so the problem surfaces at chunking time (A Dead-Letter Queue Is a Workflow, Not a Bin). |
| The process is killed part-way through a long run. | Nobody can say what completed, and the second run costs as much as the first. | Progress lived in memory and the sink was an append rather than a keyed upsert (Checkpointing). | Derive the queue by difference against the vector table, so progress is state rather than memory and a restart is just another ordinary run (Idempotent Data Pipelines). |
| A bulk re-embed and the incremental path share one rate budget. | A document edited this morning is not retrievable until the backfill finishes tomorrow. | One limiter, two workloads, no isolation — and the bulk job always has work queued, so it always wins (Bulkheads). | Separate the budgets, or admit the bulk job only outside the window where freshness is promised, and write the resulting number down as an SLO (The Freshness SLO). |
| The embedding client library is upgraded. | Newly embedded chunks retrieve slightly differently from older ones carrying the same recorded model version. | Tokenization or truncation defaults moved with no schema change and no change to the version string you record (Semantic Changes). | Put the client version into the recorded embedding version, and re-embed a fixed sample to compare before rolling the upgrade across the corpus (Data Tests). |
| Chunks are written to the chunk table while the run is in progress. | The job reports success and the index is short by exactly the rows added during the run. | No watermark, so completeness was evaluated against a set that moved (The High-Water Mark). | Record the watermark at the start, embed up to it, and let the next run take the remainder — the discipline any incremental batch already uses (Incremental Processing). |
What the run costs, and how to schedule it
The cost shape here is unusual for this domain and worth internalising, because it changes which optimisations are worth doing. In a normal data pipeline the levers are bytes scanned and bytes shuffled, and compute is a cluster you already pay for. Here the dominant term is a count of items sent to somebody else, and the only real lever is sending fewer of them.
That is why idempotency shows up in the cost section rather than only in the reliability section. A non-idempotent job that has to restart pays for the same chunks twice, and the fix costs a composite primary key. There is no other optimisation in this pipeline with a return anything like that (Compute Waste).
Scheduling follows from the same observation. Incremental work is small and wants to be fast; bulk work is enormous and wants to be cheap and interruptible. Trying to serve both from one job and one budget produces a pipeline that is simultaneously too slow for freshness and too aggressive for cost (Cost vs Freshness).
The dominant term and the only one with a big lever attached — chunk size sets the count, and idempotency sets the multiplier.
The same driver arriving all at once. It is the largest chargeable event the platform has, and it is triggered by decisions that look small in a pull request (Re-embedding).
Predictable and linear in chunk count, and one of the few costs here you can compute before doing anything.
Entirely avoidable and routinely paid, because it is invisible: nothing distinguishes a paid-for-twice chunk from a paid-for-once one.
Almost always the smallest line, and almost always the first thing somebody tries to optimise.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a corpus embedded against a metered external service, shown to establish an ordering rather than a magnitude. The teaching is the ordering: the count of items dominates, and every meaningful optimisation is a way of sending fewer of them.
Which chunks need vectors soon, and which merely need them eventually?
when The corpus is small enough that a full pass is routine, and freshness expectations are measured in hours rather than minutes.
cost Freshness for new documents is the duration of a full pass, and a backfill blocks everything. Buys one code path and no coordination.
when Documents change continuously and the corpus was fully embedded once, so ongoing work is small.
cost No plan for a re-embed, which means the first model upgrade is an unplanned project. Buys the best steady-state freshness.
when Freshness is promised on new documents while re-embeds and backfills also have to happen.
cost Two pipelines, two configurations and a rate budget to divide. Buys a backfill that cannot starve the freshness path (Bulkheads).
when Documents arrive rarely and individually, and the source system can call the pipeline synchronously.
cost Couples document authoring to the availability of the embedding service, and has no natural place to put a backfill. Buys near-immediate retrievability.
when Retrieval quality does not depend on the newest documents — reference material, archives, historical corpora.
cost Requires actually writing and monitoring the SLO, or "deferred" quietly becomes "never" (The Freshness SLO).
Batch sizes, request-rate limits, maximum input lengths, whether a batch endpoint exists at a different rate budget from the synchronous one, and how partial batch failures are reported all vary by provider and change over time. Design the job so that none of those numbers is embedded in its structure — a per-item unit of work with a keyed sink is correct under any of them — and re-read the current documentation before tuning concurrency.
How to build it
Most important first.
- Key the vector table on
(chunk_id, embedding_model_version)and write with an upsert. That single decision makes retries harmless, makes re-runs no-ops, and turns a model upgrade into an insert rather than a destructive overwrite (Upserts and Merges). - Derive the work queue by difference: select chunks with no vector at the target version and no dead-letter entry. A restart re-derives the remaining work from state, so no cursor has to survive a crash and no progress file can go stale (Full Refresh vs Incremental).
- Bound the run before it starts. Count the units of work and publish the number — a job whose size is unknown until it finishes cannot be scheduled, approved, or reasoned about while it is running.
- Handle partial batch failure per item: split the failed batch, retry members individually, and route the ones that still fail into a dead-letter table with a machine-readable reason. Never discard a batch because one member was bad (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Enforce your own concurrency limit so the service's limiter never has to enforce one for you, with backoff and jitter on the responses that tell you the limit was wrong (Without Jitter, Every Client That Failed Together Retries Together).
- Validate input length before calling. A pre-flight count turns a runtime failure into a chunking decision made at build time, which is where it belongs (Parse, Validate, Authorize, Process).
- Record an input watermark at the start of the run, so "the corpus is embedded" is a claim about a defined set rather than about a set that changed while you were reading it (The High-Water Mark).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The job guarantees that every chunk present at the recorded watermark, whose text the model accepted, has one vector at the target version. That guarantee is a property of the keyed upsert, not of the job reaching the end.
- Delivery to the embedding service is at-least-once: a timed-out request may have succeeded. Because the sink is keyed on a deterministic identity, the resulting state is effectively-once regardless of how many attempts were made — that is a property of this sink design and disappears the moment the write becomes an append (Deduplication).
- No ordering is guaranteed and none is required. Chunks are independent, which is the one genuinely easy property of this job and the reason it parallelises at all.
- It guarantees nothing about chunks created after the watermark and nothing about chunks that failed validation. Both are visible only if the run publishes counts rather than a success flag (Missing Rows).
- It does not guarantee comparability with vectors written by an earlier model version, and no storage layer will stop you from mixing the two in one index (Re-embedding).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Reconcile three counts: chunks at the watermark, minus dead-lettered chunks, must equal vectors at the target model version. Anything else means the run is under-reporting its own completeness (Reconciliation).
- It misses vectors that exist but were computed from stale chunk text. Storing the chunk text hash beside the vector and asserting it matches the current chunk row closes that hole, and is the reason the hash earns a column.
- It also cannot see a vector that is present, correctly keyed and numerically wrong — from a silently truncated input, for instance. Re-embedding a small fixed sample and comparing for equality is the cheap detector, and it is worth running after every client library upgrade (Data Tests).
- Embedding decides how quickly a newly changed document becomes retrievable, because it is the only stage in the pipeline whose throughput is set by somebody else.
- Two paths with different freshness is usually the right shape: a low-latency path for the handful of chunks that changed, and a bulk path for backfills and re-embeds, with the bulk path explicitly not competing for the same rate budget (Lambda Architecture).
- Under a single shared limit, a large backfill starves the incremental path and a document edited this morning becomes retrievable after the backfill ends. Isolating the two budgets is a freshness decision wearing a concurrency costume (Bulkheads).
- The only schema change that really matters here is the arrival of a new model version, and the table was designed for it: a new version means new rows, never modified ones (Schema Evolution).
- Vector dimensionality is part of the model version. A store that pins a dimension per collection makes a version change into a new collection, which is inconvenient and also an honest reflection of what is happening (Re-embedding).
- Client library upgrades are semantic changes with no schema footprint at all: tokenization, truncation behaviour and default parameters can move without any signature changing (Semantic Changes).
- Recovery is an ordinary re-run. A correctly designed job needs no recovery mode, because the work queue is derived by difference and the second run does exactly what the first did not (Idempotent Data Pipelines).
- An abandoned run costs only the units already paid for and leaves no partial state to clean up — provided nothing was written directly into the serving index rather than into the vector table (Atomic Publish).
- The one unrecoverable case is losing the chunk table while keeping the vectors. A vector without its text is uninterpretable and cannot be re-derived, which makes the chunk table the artefact to protect rather than the index (Backup Strategy).
What can go wrong
- A retry storm against the rate limiter, where the response to rejection is more load (Retry Storms: The Load You Generated Yourself).
- Over-length chunks skipped by an exception handler, producing a corpus with a systematic and invisible bias against long structured content.
- One enormous batch holding up a worker while the rest of the pool idles — head-of-line blocking in a job that had no reason to have any (Straggler Tasks).
- Vectors written under a version label that does not match what actually produced them, because two services read the model name from two different configuration files.
- No watermark, so completeness is asserted against a set that moved during the run (The High-Water Mark).
- The mitigation failing: a count reconciliation that treats dead-lettered chunks as complete, so a run that gave up on ten thousand documents reports a perfect match.
- "Embedding is a model concern, not a pipeline concern." The model call is one line. Everything that decides whether the corpus ends up complete — keys, watermarks, retries, limits, dead letters — is ordinary batch engineering (Orchestration).
- "Just add more workers." Past the service's limit, additional workers produce retries rather than vectors, and the retries make the limiter engage harder still (Retry Storms: The Load You Generated Yourself).
- "The job finished, so the index is complete." It is complete with respect to the watermark. Chunks written since are missing, and nothing in the exit code refers to them (The Pipeline Succeeded. The Data Is Wrong.).
- "A failed batch can be dropped and picked up next run." Only if the next run derives its work by difference. If it works from a stored cursor, the dropped batch is gone for good (Missing Rows).
- Embedding sends the full text of every chunk to an external service. That is a bulk egress of the entire corpus, and it is a classification decision rather than an engineering one — it belongs in a design review, not in a client configuration file (Data Classification, Egress Security).
- A dead-letter table holds the same content as the corpus and almost never inherits its access controls, which makes it the quietest copy of your most difficult documents (Data Access Control).
- Deleting a document has to delete its vectors at every model version, which is a second reason the version belongs in the key rather than in a column somebody remembers to filter on (Deletion Requests).
Operating it
- Units completed, units remaining and units dead-lettered, as three counters rather than one percentage. A percentage over a moving denominator is not a progress bar (Counters: The Slope Is the Signal).
- Rate-limit responses per minute, kept separate from other errors. It is the one signal that tells you whether adding workers will help or hurt (Saturation: The Reading Utilization Cannot Give You).
- Dead-letter table size with a breakdown by reason, reviewed after every run rather than merely written to (Data Incidents).
- Units of work published before the run and reconciled after it, attributed to a corpus and a reason, so a re-embed is a line item rather than a surprise (Cost Attribution).
- At ten times, the loop becomes a worker pool with a bounded queue, and the parameter that matters is the concurrency the service tolerates rather than the parallelism your cluster can offer (Sizing a Thread Pool).
- At a hundred times, the run stops fitting in one window and becomes a partitioned resumable job with per-partition progress. The failure that matters is then a partition that quietly stopped, not a process that crashed (Straggler Tasks).
- Adding workers helps up to the limiter and hurts beyond it. This is the clearest instance in the whole domain of parallelism that does not scale linearly, and the curve bends down rather than flattening (Why Eight Cores Give You Four and a Half).
- The dominant driver is units of work — chunks multiplied by the number of times you run over them. Storage, orchestration and the cluster running the loop are all small beside it, which inverts the usual data-pipeline intuition about where the money goes (What Actually Drives Data Platform Cost).
- Re-embedding a whole corpus is the largest single chargeable event this platform has, which is why chunking decisions and model upgrades deserve a design review rather than a pull request (Re-embedding).
- The cheapest optimisation is not doing the work. A keyed sink and a difference-derived queue mean an interrupted run never pays twice, and that is worth more than any batching improvement you can make (Compute Waste).
- A composite key and multiple retained model versions cost storage and a wider index. They buy idempotent retries and non-destructive upgrades, which together remove the two failure classes that make these jobs miserable.
- Holding your own concurrency below the service limit leaves throughput on the table. It buys a run whose duration you can predict and whose failure mode is slowness rather than collapse (Concurrency Limits: An Unbounded Server Is a Slower Server).
- Pre-flight length validation costs a tokenization pass over the whole corpus. It buys the absence of a silent, systematically biased gap, which is the hardest defect in this module to detect after the fact.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe job shape — keyed sink, watermarked input, work queue by difference, per-item failure handling — holds for any embedding provider and any vector store. What changes is only the name of the limit you are working against and whether the store performs the upsert natively or you emulate it.
- SCALE-SPECIFICBelow roughly a corpus that embeds inside one run, a plain loop with a keyed upsert is complete and correct, and a worker pool is unnecessary machinery. Everything about pools, watermarks and partitioned progress becomes mandatory only once a single run no longer fits in the window you are willing to hold open.
- CLOUD-SPECIFICWith a managed embedding endpoint the constraint is a request-rate limit you do not control and cannot raise on the day you need it; with a self-hosted model on your own accelerators the constraint is queue depth and memory, which you can scale but must operate. The job design is identical and the failure symptom is completely different — rejection versus timeout.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what a checkpoint has to contain for a restart to be safe when work is spread over many workers, and why deriving the remaining work from state is strictly stronger than remembering a position.
- — DevOps / Production Engineering owns the rollout of a client library upgrade whose blast radius is every vector written after it — a deployment that changes data rather than behaviour, and that needs a canary corpus rather than a canary instance.