Lambda Architecture
A batch layer that is authoritative but late, a speed layer that is fresh but provisional, and a serving layer that merges them — bought with two implementations of the same logic that must agree forever.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If the batch result is trusted and hours old while the streaming result is immediate and approximate, can a platform serve both from one interface — and what does maintaining both cost?
A consumer who needs an answer now and needs it to be right later: an operations dashboard that must show today's orders within minutes, and a finance close that must show the same day's orders exactly, from the same table, without either party knowing that two systems computed it (Who Actually Consumes This Data).
Two grains for the same concept, which is the pattern's defining property. The batch layer holds the recomputed truth for every complete period; the speed layer holds a provisional increment covering only the time since the last batch run. The serving layer must merge them into one row per period without double counting the overlap (Grain: What Does One Row Represent?).
Run the nightly batch job, and when someone asks for fresher numbers, add a streaming job that computes the same thing continuously. Point the dashboard at whichever is newer. This is how almost every Lambda platform actually starts, and it works until the two disagree.
The two layers disagree and both are defensible. Batch counted an order that the stream had not yet seen; the stream counted an order that batch later excluded as a test transaction. Nobody can say which number to publish because both jobs succeeded (Two Dashboards, Two Numbers).
- The two layers disagree and both are defensible. Batch counted an order that the stream had not yet seen; the stream counted an order that batch later excluded as a test transaction. Nobody can say which number to publish because both jobs succeeded (Two Dashboards, Two Numbers).
- A business rule changes and only one implementation is updated, because the person who changed it did not know the other existed. The divergence is invisible until the batch layer overwrites the speed layer at the next boundary and the number visibly jumps.
- The merge double counts. The batch view covers up to 04:00 and the speed view was started at 03:55, so five minutes of events are in both and the serving layer sums them (Deduplication).
- Late events land in the two layers differently. Batch recomputes the whole period and includes them; the speed layer's window already closed and dropped them. The same event is counted by one layer and not the other (Late Events).
- The batch layer stops finishing within its window as history grows, so the speed layer covers a longer and longer gap, and a component designed to be provisional becomes the one people actually read (Incremental Processing).
What is actually happening
- The motivation is historical and it is worth stating plainly, because the pattern is incoherent without it. Batch was correct but slow: recomputing from the full immutable dataset gave a result you could trust and it arrived hours later. Streaming was fast but not trusted: it held approximate state, lost it on failure, and could not easily be recomputed after a bug (Batch vs Streaming Ingestion).
- Lambda refuses to choose. The batch layer recomputes views from the entire master dataset on a schedule and is authoritative. The speed layer computes only the increment since the last batch run and is explicitly provisional. The serving layer answers a query by merging the two, and discards each speed-layer increment as soon as batch covers it.
- The property that makes this work is that the speed layer's errors are self-healing: whatever it got wrong is overwritten at the next batch boundary. That is a genuinely elegant idea and it is the reason the pattern survived — it lets you run an approximate, stateful, failure-prone stream job without that job being able to corrupt anything permanently (Stateful Stream Processing).
- The cost is structural, not incidental. The same business logic exists twice, in two engines, usually in two languages, written against two different data shapes — a bounded dataset and an unbounded stream. Nothing enforces that they agree, and the way you find out that they do not is a consumer noticing a jump (Determinism: Same Input, Same Output?).
- The merge in the serving layer is its own problem. It must know exactly which period each layer covers, exclude the overlap, and handle the case where the batch run is late — at which point the speed layer is covering a longer window than it was sized for (Atomic Publish).
- Lambda is not a streaming pattern with batch attached, nor the reverse. It is a commitment to run both paths permanently, and it should be evaluated as the cost of a second permanent implementation rather than as an incremental feature.
Why anyone built two of everything
Lambda is easy to criticise and hard to understand out of context. It was designed for a period when the batch engine and the stream engine could not be the same system, and when a long-running stateful stream job was genuinely not something you would trust a quarterly financial figure to. Given those two facts, running both and letting the trustworthy one overwrite the fast one is not a compromise — it is the correct answer.
The elegance is in the direction of the overwrite. The speed layer is allowed to be wrong, because everything it produces has an expiry date: the next batch run recomputes the same period from the immutable master dataset and replaces it. That means an approximate aggregate, a lost state store or a dropped late event costs you accuracy for a few hours and costs you nothing permanently (Streaming State).
What the diagram does not show, and what decides whether a Lambda platform is pleasant or miserable to run, is that the two paths compute the same rule. That is a human commitment enforced by tests, and the pattern gives it no structural support whatsoever.
Where one event actually lands
The merge is where Lambda is won or lost, and the fastest way to understand it is to follow individual events across a batch boundary. Two things can go wrong and both are invisible in any per-layer monitoring: an event counted twice because both layers claim its period, and an event counted zero times because the speed layer's window closed before it arrived and the batch run had already passed it.
The timeline below uses a batch run that covers everything up to 10:00 and a speed layer responsible for 10:00 onward. Follow event e3 in particular: it happened at 09:58 and arrived at 10:06, after the batch boundary. Whether it is counted depends on whether the batch layer reprocesses the 09:00–10:00 period on its *next* run, and whether the speed layer's lateness policy admits an event whose event time is before its own coverage window (Late Events).
This is why "the two layers must agree" is not really about business logic. Two implementations can encode identical logic and still disagree, because they disagree about which events belong to which period. Aligning the lateness policy and the period definition between the layers is as important as aligning the aggregation rule, and it is much more often skipped (Watermarks).
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| e1 | 09:12 | 09:13 | Batch-authoritative period The straightforward case: happened and arrived well inside the period batch recomputes. |
| e2 | 09:57 | 09:58 | Batch-authoritative period Also counted by the speed layer while it was live, then expired when the batch view took over. Correct only because the serving layer excluded the overlap. |
| e3 | 09:58 | 10:06 | Batch-authoritative period, on the next run only The late event. Batch will include it when it next recomputes 09:00-10:00; the speed layer's window for that period has closed. Between now and that run, this event exists nowhere. |
| e4 | 10:03 | 10:04 | Speed-layer coverage Provisional. It will be recounted by batch at the next boundary and the speed increment discarded. |
| e5 | 09:59 | 10:12 | Dropped by the speed layer Beyond the allowed lateness for a window that has already emitted. If batch does not reprocess the period, this event is lost from the merged answer entirely. |
Clock labels on a teaching timeline, not measurements. The lesson is e3 and e5: two events that differ only in arrival time end up in completely different places, and neither layer reports an error about either.
The bill: one rule, two implementations, forever
The maintenance cost of Lambda is not the two jobs. It is the obligation that the two jobs encode the same rule, in perpetuity, across every change either of them undergoes — and the fact that nothing in the architecture enforces it.
The mitigation that actually works is a differential test: take a fixed historical window, run both implementations over it, and assert the outputs match record for record. It is cheap to build and it catches the majority of divergences before they reach a consumer. It also has a specific blind spot worth stating: run over clean historical data, both implementations look identical, and they diverge precisely on the paths a replay does not exercise — late arrivals, restarts, state loss and out-of-order events (Data Tests).
Before adopting the pattern, price the alternative honestly. If your engine can express both paths in one dialect, most of the duplication disappears and Lambda becomes cheap. If your inputs are genuinely a replayable log and your transformations are stream-expressible, Kappa removes the duplication entirely and charges you elsewhere. Lambda is the right answer when neither of those is true, which is a narrower set of circumstances than it was (Batch and Streaming Unification).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A business rule is changed in one layer only. | The number visibly jumps at the batch boundary, in a consistent direction. | Two implementations, one changed. Nothing enforces or even reports the difference (Semantic Changes). | Differential test in the build, plus a divergence metric on the most recent completed period so an unreported change is caught within one cycle. |
| The batch run is late or fails. | Numbers for a completed period stay provisional; accuracy quietly degrades exactly when someone is closing a report. | The speed layer's coverage window is defined as "since the last successful batch run" and has silently stretched (When a Task Fails Mid-DAG). | Alert on coverage-window length rather than only on task failure, and expose the provisional/authoritative status to consumers on the number itself. |
| The overlap between layers is computed from timestamps. | Small, persistent over-counting near every boundary that nobody can reproduce on demand. | Two systems' clocks and period definitions do not agree closely enough for a timestamp comparison to partition records cleanly (Processing Time). | Merge on record identifiers with an explicit recorded coverage range per increment, never on "greater than the batch high-water timestamp". |
| History grows past the batch window. | The batch job starts overrunning; someone makes it incremental to fit. | Full recomputation scales with history, not with the increment (Full Refresh vs Incremental). | A legitimate fix that changes the pattern: an incremental batch layer no longer recomputes from source, so the property that made it authoritative must be re-established by reconciliation instead. |
| The differential test passes for a year and a divergence still ships. | The layers agree on all historical data and disagree in production. | The mitigation failed: the test window contained no late arrivals, no restarts and no out-of-order events, which is where the implementations actually differ. | Build the test window from a recorded production period including its late and duplicate traffic, rather than from a clean synthetic replay (Late-Arriving Data). |
The batch job is SQL in the warehouse; the speed layer is application code in a stream processor. Each is owned by whoever wrote it, each is changed when its own consumer complains, and agreement is verified by a person comparing dashboards.
Express the rule once wherever the engines permit — a shared definition, a shared library, or a single dialect that both paths execute. Where they cannot share code, keep a differential test that runs both over the same historical window and fails the build on any mismatch, and include late-arrival and restart scenarios in that window rather than only clean data.
Divergence between the layers is not detectable from either side. Each job succeeds, each produces plausible output, and the only observer positioned to notice is a consumer comparing numbers across a batch boundary. An assertion that compares the two directly is the only mechanism that turns a silent semantic failure into a loud build failure.
How to build it
Most important first.
- Only adopt it if the freshness requirement is real. Establish first that a decision genuinely changes based on data younger than the batch interval; most dashboards demanding minutes are read once a day, and a Lambda platform built for one of those is a permanent tax for no benefit (Cost vs Freshness).
- Make the immutable master dataset the single input to both layers. If batch reads the warehouse and the speed layer reads the log, they are not computing the same thing from the same facts and divergence is guaranteed rather than possible (Keeping Raw History: The Recovery Position and the Liability).
- Express the shared business logic once wherever the engines allow it — a shared SQL definition, a shared library, a unified batch-and-streaming API. Where they do not allow it, write a differential test that runs both implementations over the same historical window and asserts the outputs match (Batch and Streaming Unification).
- Give the merge an explicit, recorded boundary. Each speed-layer increment should carry the exact period it covers, and the serving query should exclude anything the batch view already includes rather than relying on timestamps agreeing across systems.
- Publish the provenance to the consumer. A number should be able to say "this period is batch-authoritative" or "this period is provisional", because a consumer who knows a figure is provisional behaves differently from one who does not (Dataset Documentation).
- Monitor divergence continuously rather than discovering it at the boundary: for the most recent complete period, compare what the speed layer said against what batch subsequently computed, and alert on a gap beyond a stated tolerance (Quality Alerting).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The batch view guarantees a recomputed-from-source result for every complete period, which is the strongest correctness guarantee available in this domain — provided the master dataset is immutable and the job is deterministic (Reprocessing vs Retrying).
- The speed view guarantees only freshness. It is explicitly provisional: approximate aggregates, incomplete handling of late events and lossy state after failure are all acceptable by design, because batch will overwrite it.
- The serving layer guarantees a single answer per query. It does not guarantee that the answer is stable over time — the same query for the same period legitimately returns a different number before and after the batch boundary.
- Nothing guarantees the two implementations encode the same rule. That is the guarantee people assume exists and it is the one that has to be built with tests (Data Tests).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that defines this pattern is layer divergence: once a period becomes batch-authoritative, compare the batch result against what the speed layer had been reporting for that same period, and alert when the gap exceeds a stated tolerance (Reconciliation).
- Add an overlap check on the merge: assert that no source record contributes to both the batch view and a live speed increment for the same period. This is the check that catches double counting, and it must be expressed on record identifiers rather than on timestamps (Duplicate Rows).
- Both miss the case where the two implementations share a wrong rule — they will agree perfectly and both be wrong. Only a reconciliation against the source system catches that, and it is the check to build before either of the two above (The Pipeline Succeeded. The Data Is Wrong.).
- The consumer experiences a two-part freshness: complete periods are as fresh as the last batch run, and the current partial period is as fresh as the speed layer's processing lag. Presenting that as one number misleads whichever half is worse.
- The speed layer's coverage window is not fixed — it is "since the last successful batch run". A failed batch run silently extends it, which is exactly when the provisional layer is least equipped to be accurate (When a Task Fails Mid-DAG).
- Because the batch layer overwrites, freshness and stability trade against each other directly: a consumer reading the current period gets an answer that will change, and a consumer waiting for the batch boundary gets one that will not (The Freshness SLO).
- Every business-rule change is two changes, in two codebases, that must ship close enough together that the divergence window is tolerable. Treat them as one atomic change with two artefacts, not as a change and a follow-up ticket.
- A schema change upstream hits both layers, and they usually fail differently: batch fails loudly at read time while the stream job may absorb the change silently or crash-loop. Testing a schema change against both paths is non-optional here (Schema Evolution).
- The most dangerous change is one that alters what a *period* means — a timezone shift, a change in which timestamp defines the period, a redefinition of when an order counts. The two layers may adopt it at different moments and the overlap logic breaks at exactly that boundary (Event Time).
- Recovery from a batch-layer bug is a re-run over the affected range, which is the pattern's strength: the batch view is defined as a pure recomputation from the master dataset, so a corrected run simply overwrites it (Planning a Backfill).
- Recovery from a speed-layer bug is usually to do nothing. It is provisional and it will be overwritten at the next boundary; the correct action is to fix the code and let batch heal the history, not to backfill the speed layer.
- Recovery from a *merge* bug is the hard one, because the published number was correct in each layer and wrong in combination. Re-publishing the serving view for the affected periods requires knowing exactly which increments were included, which is why the merge boundary must be recorded rather than inferred (Atomic Publish).
- If the batch layer misses several runs, the speed layer is covering a window it was not sized for and its state may not fit. Plan the catch-up explicitly: run batch forward through the missed periods before trusting the merge again (What Backfills Break).
What can go wrong
- Silent logic divergence between the two implementations, discovered by a consumer when the number jumps at a batch boundary.
- Double counting in the overlap between the batch coverage window and the live speed increment.
- Late events included by batch and dropped by the speed layer, producing a systematic, direction-consistent gap that looks like a business trend (Late Events).
- A batch run that fails or is late, silently extending the provisional window at the moment accuracy matters most.
- The batch layer outgrowing its window as history accumulates, so the pattern quietly becomes streaming-only with a batch job that never finishes (Full Refresh vs Incremental).
- The mitigation failing: a differential test that runs both implementations over a fixed historical window, passes forever, and does not exercise the late-event and failure paths where the implementations actually differ.
- "Lambda is the safe, mature choice." It was a genuine answer to a real constraint and it carries the highest permanent maintenance burden of any pattern here. Neither Lambda nor Kappa is the simpler one in general; they are simpler under different conditions (Kappa Architecture).
- "The speed layer is a temporary approximation, so its bugs do not matter." Its bugs are what consumers read for the entire current period, which for a daily batch is most of the working day.
- "If both layers are green, the numbers agree." Both layers succeeding says only that two programs ran. Agreement is a separate assertion and needs its own test (Data Tests).
- "We can drop the batch layer once streaming is reliable." That is a real option and it is called Kappa, and it requires the log to retain everything you might reprocess and the stream job to replay history at a rate the batch layer used to manage. It is a different trade, not a removal of one (Replay from the Log).
- "Lambda is about batch versus streaming." It is about *reconciling* the two into one answer. The merge and the divergence monitoring are the pattern; the two jobs are just its inputs.
Operating it
- Divergence between layers for the most recently completed period, as a signed percentage of the batch value, tracked over time. A drift that grows is a logic change in one implementation; a constant offset is usually a late-event or overlap issue (Quality Alerting).
- Batch run duration against the batch interval, with headroom shown. This is the leading indicator that the pattern is about to stop working (Pipeline Metrics).
- Speed-layer coverage window length — how much wall-clock time the provisional increment is currently responsible for. It should be flat; every spike is a batch failure (Freshness Monitoring).
- The number of records counted by both layers for the same period, which should be zero and is the direct measure of the merge's correctness.
- At 10x volume the batch layer is the first thing to break, because a full recomputation grows with history rather than with the increment. The usual response is incremental batch, which is sensible and quietly changes what the pattern guarantees.
- At 10x metric count, the duplicated-implementation cost multiplies directly: every metric is two implementations, and the differential test matrix grows with it.
- At 10x consumer count the merge semantics become the problem, because more consumers means more people who did not know the current period was provisional (Dataset Documentation).
- Nothing about the pattern gets cheaper at scale. This is a rare case where the cost curve has no economies to offer, which is a large part of why the field moved on where it could (Kappa Architecture).
- Engineering time is the dominant cost and it is permanent: two implementations, two sets of tests, two deployment paths, two on-call surfaces, and a differential test that has to be maintained (Cost Attribution).
- Compute is paid twice for the recent window by construction — the speed layer computes it live and batch recomputes it later. That duplication is the pattern rather than a waste to be optimised away (Compute Waste).
- The batch layer's cost scales with total history if it fully recomputes, which is the classic driver of the pattern outgrowing its window; making it incremental reduces cost and weakens the "recomputed from source" guarantee that justified it (Full Refresh vs Incremental).
- Continuous speed-layer compute is paid whether or not events are flowing, which makes low-volume Lambda pipelines disproportionately expensive relative to what they deliver (Cost vs Freshness).
- You buy a fresh answer and a correct answer from one interface, and you pay for it with a permanent second implementation of every rule.
- The self-healing property that makes the speed layer safe to run approximately is the same property that makes published numbers unstable — a consumer's figure can legitimately change overnight.
- Making batch incremental to keep it inside its window is the standard fix, and it trades away the "recompute everything from the master dataset" guarantee that made the batch layer authoritative in the first place.
Lambda merge lab
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe three-layer structure and the duplicated-logic cost are properties of running two computation paths over the same facts, and they hold regardless of which engines fill the layers or whether the storage is a warehouse or a lake.
- ENGINE-SPECIFICThe size of the duplication cost depends entirely on whether one engine can express both paths. Where batch and streaming share a runtime and a dialect, the two implementations can share most of their logic; where they are separate systems with separate languages, the duplication is total and the differential test is the only safeguard.
- SCALE-SPECIFICThe pattern is most defensible where the batch layer comfortably recomputes all history within its interval. Once history outgrows the window the batch layer must become incremental, which removes the recompute-from-source property that made it authoritative and changes the pattern into something that needs re-justifying.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the reason the speed layer was ever considered untrustworthy: what a stateful stream operator loses on failure, what a checkpoint actually restores, and why a recomputation from an immutable input is a stronger claim than an incrementally maintained one.
- — DevOps / Production Engineering owns shipping a rule change to two systems as one atomic release, which is the delivery problem that makes or breaks this pattern in practice.