Batch vs Streaming Ingestion
Not old versus new. Two designs with different freshness shapes, different failure surfaces, different recovery stories and very different operational burdens.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which of these two should this particular source use, and what is the criterion that decides it rather than the preference that usually does?
The consumer decides this and almost nobody asks them. The question to answer is: what decision does this data drive, how often is that decision made, and what would it cost to make it on data that is one interval old? A finance close, a fraud rule and a weekly cohort report give three different answers, and the platform frequently gives all three the same one.
The grain differs between the two in a way that outlives the ingestion layer. Batch delivers a row as of a read, so intermediate states between runs are lost. Streaming delivers an event per change, so intermediate states are preserved. That is a modelling difference, not a latency one, and it constrains which questions the warehouse can ever answer (Event vs Snapshot Modeling).
Pick one for the whole platform. Usually streaming, because it is described as the modern option, sometimes batch because it is what the team knows. Either way the decision is made once, architecturally, and applied to every source regardless of what any consumer needs.
A daily finance report is fed by a streaming pipeline. It is read once a morning, so the freshness is worth nothing, and the team now carries lag alerting, rebalance behaviour and a retention cliff for a number that changes once a day (Streaming Ingestion).
- A daily finance report is fed by a streaming pipeline. It is read once a morning, so the freshness is worth nothing, and the team now carries lag alerting, rebalance behaviour and a retention cliff for a number that changes once a day (Streaming Ingestion).
- A fraud check is fed by an hourly batch. The freshness is a real product constraint and the design cannot meet it at any interval, because per-record latency is bounded below by the window (Batch Ingestion).
- The platform is "streaming" but the transformation runs nightly, so every consumer experiences nightly data while the organisation believes it is real-time and makes commitments accordingly.
- Batch was chosen for simplicity and the interval was shortened repeatedly to chase freshness, until runs overlap, files are tiny and the pipeline has all of streaming's costs with none of its properties.
- Streaming was chosen for freshness and nobody sized the landing buffer, so the lake fills with tiny files and every analytical query is slower than the batch version it replaced (File Size and the Small-Files Problem).
- A team migrates a working batch pipeline to streaming for consistency, discovers that state reconstruction from a change stream is a genuinely harder modelling problem than reading current state, and ships a subtly wrong dimension table (What a CDC Event Contains).
What is actually happening
- The two differ on what the unit of work is. Batch's unit is a window: bounded, named, retryable, publishable atomically. Streaming's unit is a position: continuous, per-partition, advanced rather than completed. Every downstream difference follows from that one.
- They therefore fail differently in kind rather than in degree. A batch failure is discrete and visible — a window is missing. A streaming failure is continuous and gradual — lag grows. Discrete failures are easier to detect and harder to recover if the schedule moved past them; gradual failures are easier to recover and harder to notice before a threshold (Ingestion Failure & Recovery).
- Recovery is genuinely asymmetric. Batch recovers by re-reading the source, which is bounded by what the source retains and costs the source load. Streaming recovers by replaying the log, which is bounded by broker retention and costs the source nothing. Neither is universally deeper (Replay from the Log).
- Operational burden is the axis most often left out and it dominates the real decision. Streaming adds a permanently running stateful process, consumer-group semantics, partition sizing, poison-message handling and a retention cliff. Those are not incidental — they are the design (Data Platform Engineering).
- Freshness is a *shape*, not a number. Batch gives a sawtooth bounded by the interval; streaming gives a continuous curve bounded by lag. A consumer reading once a day cares about neither shape and only about whether the data was complete when they read it (Cost vs Freshness).
- The two also differ on who absorbs backpressure. In batch, a slow consumer simply reads later — the data is at rest. In streaming, a slow consumer accumulates lag against a retention deadline, so backpressure is a live concern with a losing end state (Backpressure).
Six axes, and the two everybody argues about
Comparisons of these two usually run on one axis — latency — and occasionally on a second, cost. Those are the two least decisive. Latency is usually irrelevant because the consumer's decision cadence is slower than either design, and cost is genuinely close once the continuous compute of one is weighed against the repeated source reads and orchestration overhead of the other.
The axes that actually decide are the ones below it: what the failure looks like, how deep recovery goes, how much operational surface the team takes on, and whether the history of individual changes is preserved. The last of these is the one people discover late, because it is not a property of ingestion at all — it is a constraint on what the warehouse can ever be asked.
Read the table as a set of questions to answer about your own source, not as a scoreboard. Several rows have no better side; they have a side that matches the situation.
| Axis | Batch | Streaming | What decides it |
|---|---|---|---|
| Freshness shape | Sawtooth, bounded by the interval. Worst case is a full interval old. | Continuous, equal to lag plus landing buffer. Degrades smoothly under load. | The consumer's decision cadence. If no decision is made faster than the interval, this row is a tie at zero value. |
| Failure shape | Discrete and visible: a window is present or missing. | Gradual and quiet: lag grows with no threshold until retention. | Which kind of failure the team can detect. Discrete failures suit grid dashboards; gradual ones need thresholds derived from retention. |
| Recovery depth | Re-read the source. Bounded by source history, costs source load, returns current values for old rows. | Replay from offset. Bounded by broker retention, costs the source nothing, returns exactly what happened. | Whether the source retains history and whether replay depth exceeds it. Neither is universally deeper. |
| Operational burden | A schedule, a bookmark, a lock. No long-lived state. | A running stateful consumer, group membership, rebalances, partition sizing, poison messages, a retention cliff. | The team. This is the axis most often omitted from the comparison and most often decisive in practice. |
| History preserved | State at read time only. Changes between runs are lost forever. | Every change the producer emitted, in per-partition order. | Whether any consumer needs to know what happened between two observations. Irreversible once chosen (Event vs Snapshot Modeling). |
| Independent consumers | Each consumer re-reads the source or reads the landed copy. Adding one adds source load. | Many consumers read the same log at their own positions, with no coordination and no extra source load. | How many teams want the same data. This is the strongest argument for a broker and it has nothing to do with latency (Kafka as a Log, Not a Queue). |
| Cost shape | Bursty; zero when idle; grows with run frequency times source read cost. | Continuous; paid while idle; grows with retention and consumer capacity. | Event rate and idle fraction. A low-volume source on a stream pays to wait. |
The difference that outlives the ingestion layer
Freshness and cost are reversible decisions. You can change a schedule, resize a buffer, add consumers. The choice that is not reversible is what the ingestion captured, because a change that was never observed cannot be recovered later from a source that has moved on.
A batch extract observes state. If an order went pending → paid → refunded between two runs, the warehouse sees refunded and has no way to know the other two happened. Every question of the form "how long do orders stay pending" and "how many orders were ever paid" is unanswerable, and — this is the part that bites — it is unanswerable *silently*. The queries run. They return plausible numbers that mean something narrower than the person asking believes.
A change stream observes transitions. Reconstructing state from it is more work and it is work you can choose to do; recovering transitions from state is not work, it is impossible. That asymmetry is the strongest non-latency argument in this comparison, and it is the one most often discovered a year after the decision (What a CDC Event Contains).
The grain table below is the practical form of this. Track what one row means at each stage of each design, because a metric written against one grain and pointed at the other is the most common way this decision produces a wrong number.
| Stage | One row is | Breaks if |
|---|---|---|
| Source table | One entity, in its current state. Identical for both designs — this is what both are reading from. | You assume it holds history. It holds the latest value and has forgotten the rest. |
| Batch extract output | One entity as it appeared at one read time. The same entity appears once per window in which it changed. | You count rows and call them changes. Three changes inside one window produce one row, and the count of "changes" is really a count of "windows in which something changed". |
| Streaming ingest output | One change to one entity, as the producer modelled it. | You count rows and call them entities. An entity with five changes is five rows and one entity (Grain: What Does One Row Represent?). |
| Batch staging model | One entity, latest observed state, chosen by the newest window it appeared in. | Two windows are processed out of order, so an older observation overwrites a newer one — which is why "latest" must be chosen by a source column, not by processing order. |
| Streaming staging model | One entity, reconstructed as the latest change per key. | "Latest" is chosen by arrival or offset rather than by a source-provided version, so a cross-partition reordering produces a stale current state (CDC Ordering and Transaction Boundaries). |
| Fact table, either design | One business event at a declared grain. | The two paths feed the same table with different deduplication semantics, so the same real-world event is one row via one path and two via the other (Duplicate Rows). |
The two designs converge at the fact table and mean different things everywhere before it. A hybrid platform that does not make the grain explicit at each stage will eventually double-count exactly the entities that both paths touched.
Pick batch or streaming as a platform standard, implement it for every source, and find out eighteen months later that a question about state transitions cannot be answered — or that a team is carrying broker operations for a dataset read once a day.
Ask two questions per source. What is the fastest decision anyone makes on this data, and does anyone need to know what happened between two observations? The first sets the freshness requirement; the second decides whether state-at-read-time is sufficient. Then check what shape the source is: a push source wants streaming regardless, a modest queryable table is happy with batch.
Freshness can be changed later by adjusting an interval or adding consumers; captured history cannot be changed later at all, because the transitions that were not observed no longer exist anywhere. Deciding on the reversible axis and discovering the irreversible one afterwards is how platforms end up with a modelling constraint nobody chose.
Making the call, and what each answer costs
In practice the decision resolves quickly once it is asked in the right order: source shape first, then history requirement, then consumer multiplicity, then freshness, and operational capacity as a veto over all of them. Cost rarely decides it and latency decides it less often than anyone expects.
Source shape first because it is the least negotiable. A webhook, an event stream or a database change log is push-shaped, and wrapping it in a scheduled pull means buffering somewhere — which is where events are lost by a component nobody designed to be durable (Inbound Webhooks).
Operational capacity as a veto because a design a team cannot run is not a design. A streaming pipeline whose lag alerts nobody understands will lose more data at the retention cliff than a nightly batch would have lost in a year of skipped windows.
And note the option that is missing from most versions of this debate: run both, for different sources, with the same raw conventions. That is what most mature platforms actually look like, and the coherence they have is in the landing layout and the check vocabulary rather than in the cadence.
Scales with run frequency times table size when the predicate is unindexed, and with change volume when it is indexed. Paid by the source owner.
Scales with frequency times source count and is invisible until a platform is running hundreds of thousands of small tasks a day.
Paid while idle. The dominant cost for low-volume topics and the reason a rarely-changing source is a poor fit for a stream.
Event volume times retention window. It is a recovery budget denominated in storage, and it should be argued as a recovery decision.
The convergent failure. Frequent batch and small streaming buffers produce the same problem, and it is paid by every downstream query forever (File Size and the Small-Files Problem).
Lag alerts, rebalances and partition sizing on one side; window continuity, overlap and catch-up storms on the other. Real, recurring, and absent from every infrastructure comparison.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights within each design, given to show where attention should go rather than to compare the two totals — a cross-design cost comparison is only meaningful with a specific volume, cadence and source attached, and any general claim that one is cheaper is a claim without those.
What shape is the source, what does the consumer decide, and what can the team run?
when The source is a queryable database of modest size, the consumer decides daily or slower, and no one needs intermediate states.
cost A freshness floor equal to the interval and a late-data boundary at every window edge. Buys the simplest correctness and repair story available: an explicit window, re-runnable by parameter (Batch Ingestion).
when Freshness of tens of minutes matters, the extract comfortably fits well inside the interval, and the source can absorb the read frequency.
cost More source reads, more files, more orchestration tasks. Watch extract duration against interval — when they converge, the answer is a different design, not a smaller number.
when The source is already push-shaped, or many independent consumers need the same data, or per-change history is required.
cost A continuously running stateful consumer, retention as a recovery deadline, and an actively managed landing-buffer trade between freshness and file size (Streaming Ingestion).
when The source is a database whose query load matters, deletes must be captured, or commit ordering is needed to reconstruct state correctly.
cost Replication configuration on a database you may not own, plus a connector whose failure is silent until it passes retention (Change Data Capture).
when Consumers genuinely differ — a fraud path and a finance path in the same company.
cost Two failure vocabularies, two check sets, and one shared risk: the same event arriving by both paths and being counted twice. Mitigated by identical raw conventions and one deduplication key (Deduplication).
when One source, small volume, a read replica and a scheduled query already answer every question asked.
cost Nothing, and it is frequently the right answer. Keep raw event history in the meantime, because that is the part that cannot be recreated once a question arrives that needs it (Keeping Raw History: The Recovery Position and the Liability).
How to build it
Most important first.
- Choose per source and per consumer, never per platform. The right platform has both, and the coherence people seek from picking one is bought by consistent conventions — explicit windows, keyed idempotent writes, immutable raw — not by uniform cadence.
- Start from the consumer's decision latency. If no decision is made faster than daily, ingestion faster than daily is buying nothing and costing operations. Write that answer down so the next person does not relitigate it (The Freshness SLO).
- Choose streaming when the source is genuinely push-shaped — an event stream, a webhook, a change log — because wrapping a push source in a pull interface means buffering, and the buffer is where events are lost (Ingestion Sources).
- Choose streaming when many independent consumers need the same data at their own pace. The multi-consumer replayable log is a stronger argument for a broker than latency ever is (Kafka as a Log, Not a Queue).
- Choose batch when the source is a queryable database of modest size, the consumer is periodic, and the team's operational capacity is better spent elsewhere. This is most sources in most companies and it is not a compromise — but never chase freshness by shortening the interval past the point where runs approach their own duration, because that threshold is a signal to change design rather than to change a number (Batch Ingestion).
- If you run both, make them produce the same raw layout — same partitioning by arrival, same file conventions, same event ids — so the transformation layer cannot tell which produced a given partition (The Raw Landing Zone).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Both are at-least-once in practice. Batch achieves effectively-once through window-scoped replacement; streaming through keyed idempotent writes with offsets committed after durability. Neither gets it from the transport (At-Least-Once Delivery).
- Batch guarantees no ordering at all within a window. Streaming guarantees ordering per partition. If your model reconstructs state from changes, only the second is usable, and only if you keyed by the entity (CDC Ordering and Transaction Boundaries).
- Batch offers a natural atomic publish boundary — the window. Streaming has no natural boundary and one must be constructed, usually as a landing interval that behaves like a small window (Atomic Publish).
- Batch preserves only the state at read time; changes between runs are lost. Streaming preserves every change the producer emitted. This is a completeness difference about *history*, not about rows, and it is irreversible in the batch direction (Keeping Raw History: The Recovery Position and the Liability).
- Neither guarantees completeness against the source. Both require reconciliation, and the reconciliation query is nearly identical (Reconciliation).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Batch checks window continuity, per-window counts against history, and uniqueness within a window. Streaming checks offset continuity per partition, lag against retention, and duplicate rate after landing.
- The check both need identically is reconciliation against the source for a closed period. It is the only one that observes completeness rather than internal consistency, and neither design provides it for free (Reconciliation).
- The batch checks miss a window that ran with a shifted boundary. The streaming checks miss an event a producer never published. Both blind spots are outside the pipeline, which is the general shape of what ingestion checks cannot see (The Dual Write Problem).
- A hybrid platform needs both sets and frequently has neither, because "we have monitoring" is satisfied by whichever set was built first (The Pipeline Succeeded. The Data Is Wrong.).
- Batch: sawtooth. Worst case equals the interval, average equals half of it, and a consumer reading just before a run gets the worst case. Quoting the average to a consumer describes a moment they may never read at.
- Streaming: continuous. Staleness equals consumer lag plus landing buffer, and it responds to load rather than to a clock, degrading smoothly under pressure instead of stepping.
- Both are floored by whatever comes after. A streaming ingest into an hourly transformation gives hourly freshness, and the ingestion cadence is invisible to every consumer downstream of the slowest hop (The Data Loop).
- The distinction that survives every interval change: batch cannot give per-record latency below its window, because a record must wait for its window to close. Streaming can. If a single record's latency is a product requirement, that is a structural argument rather than a tuning one.
- A schema change in batch lands inside a window and history is heterogeneous by window. In streaming it lands mid-partition with producers and consumers deployed independently, which is why schema registries exist on the streaming side and not usually on the batch one (Schema Registry).
- Migrating batch to streaming requires stitching a snapshot to a stream at a consistent position. The seam is a single point in time where a gap or a duplicate is easy to create and hard to notice later (Snapshot and Stream: the Bootstrap Problem).
- Migrating streaming to batch is easy operationally and lossy semantically: the intermediate states that the stream captured stop being captured, and any model that depended on change history quietly begins to sample instead (Event vs Snapshot Modeling).
- Changing cadence in either direction makes periods either side non-comparable for any metric sensitive to sampling. That deserves a note attached to the dataset, not just to the pipeline (Semantic Changes).
- Batch: re-run a window. Bounded, parameterised, and it re-reads the source — so recovery depth is the source's history and recovery cost is source load.
- Streaming: replay from an offset. Bounded by retention, costs the source nothing, and reprocessing a week is a configuration change rather than a project (Retention and Replay).
- The cliff is on the streaming side and the ceiling is on the batch side. Streaming recovery is excellent inside retention and impossible outside it; batch recovery is always possible but always costs the source and always returns current values for historical rows (What Backfills Break).
- Both are entirely dependent on the raw landing zone once their own recovery path is exhausted, which is the strongest argument for keeping raw immutable regardless of which design you chose (The Raw Landing Zone).
What can go wrong
- Batch: a skipped window nothing ever re-requests; overlapping runs producing duplicates; a non-atomic load exposing a partial period.
- Streaming: lag crossing retention; an offset committed before durability; a poison message blocking a partition; a small-file accumulation that degrades every downstream query.
- Shared: at-least-once duplicates, source schema drift, a bookmark or offset that advanced past data that never landed.
- The hybrid-specific one: two paths into the same table with different deduplication semantics, so the same event is counted once via one path and twice via the other (Deduplication).
- The mitigation fails in both: an alert threshold chosen as a round number rather than derived from the retention or the interval it is protecting, so it fires either constantly or never.
- "Streaming is more modern, so it is better." Modernity is not an engineering property. Streaming is better when the source is push-shaped, when many consumers need independent reads of the same data, when per-record latency is a product requirement, or when per-change history matters. It is worse when none of those hold, because it costs a continuously running stateful component and a retention cliff in exchange for freshness nobody asked for.
- "Batch is what you do before you can afford streaming." Batch is what you do when the unit of work is naturally a window, which is true of most analytical loads. Discrete, retryable, atomically publishable units are a genuine engineering advantage and not a limitation (Batch Ingestion).
- "We are streaming, so the data is real-time." End-to-end freshness is the slowest hop. Streaming ingestion into an hourly model gives hourly data, and describing the platform by its fastest component misleads everyone who plans around it.
- "Streaming is more expensive." Sometimes. It costs continuous compute and retention; batch costs repeated source reads and, at high frequency, more orchestration than anyone expects. Neither dominates in general and the comparison is only meaningful with a volume and a cadence attached.
- "You can convert one into the other by changing the interval." Shortening a batch interval far enough gives you streaming's costs — file count, orchestration overhead, overlapping runs — without its properties: no replay, no per-change history, no independent consumers.
- "Pick one for consistency." Consistency worth having is in the raw layout, the idempotency convention and the check vocabulary. Uniform cadence is consistency of the cheapest kind, bought by giving some consumer the wrong answer.
Operating it
- For batch, a window completion grid — one cell per expected window. A hole is instantly visible in a way no success rate is (Pipeline Observability).
- For streaming, per-partition lag in time with retention drawn on the same axis. Anything else understates how close a lag problem is to becoming a loss problem (Depth Is Not an Emergency; Age Is).
- For both, end-to-end freshness measured at the *serving* dataset rather than at the ingestion layer, because that is the only number a consumer experiences (Freshness Monitoring).
- For a hybrid platform, one freshness view covering both, so the answer to "how current is this table" does not depend on knowing which ingestion path fed it.
- At 10x volume, batch hits its window and must become chunked or log-based; streaming hits its partition count and must be re-partitioned. Both are foreseeable and both are usually addressed late.
- At 100x, batch's source load becomes the binding constraint and the read must move to the change log, at which point the design is streaming in all but name (Change Data Capture).
- At 100x consumers rather than volume, streaming's advantage is decisive for a reason unrelated to latency: many independent readers of the same durable log at their own positions is something batch has no equivalent for (Consumer Groups and the Parallelism Ceiling).
- At small scale, neither is under pressure and the correct criterion is entirely operational: which one can this team run well at three in the morning.
- Batch cost is bursty and proportional to run frequency times source read cost. It is zero when idle, which for a low-volume source is the dominant consideration.
- Streaming cost is continuous: consumer compute runs whether or not events flow, and broker storage is paid for the retention window regardless of whether replay is ever used (Compute Waste).
- The hidden cost on both sides is file count. Frequent batch and small streaming buffers converge on the same small-file problem, which is paid by every downstream query for as long as the files exist (File Compaction).
- The largest cost of streaming is usually not infrastructure at all but the engineering attention consumed by lag alerts, rebalances, partition sizing and schema compatibility — real, recurring, and absent from every cost comparison (What Actually Drives Data Platform Cost).
- Batch trades a freshness floor and a late-data boundary condition for operational simplicity: no long-lived state, no rebalancing, no retention cliff, and a repair story that is a re-run with a parameter.
- Streaming trades a permanently running stateful component, a retention cliff and an actively managed file-size problem for continuous freshness, per-change history, replay-based recovery and independent multi-consumer reads.
- Running both trades a single mental model for the ability to serve consumers with genuinely different needs — and costs two sets of checks, two failure vocabularies and one more place for the same event to be counted twice.
- Choosing on cadence alone trades a correct answer for a fast one. The axes that decide it more often are the source's shape, the number of independent consumers, and whether per-change history is needed at all.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe comparison axes — freshness shape, failure shape, recovery depth, operational burden, history preservation, consumer multiplicity — apply regardless of tooling. Only the magnitudes on each axis are product-specific.
- ORG-SPECIFICOperational burden is measured in the capacity of a specific team, not in the abstract. A platform group with existing broker expertise and on-call rotation pays far less for streaming than a two-person data team who would be learning consumer-group rebalancing during an incident.
- SCALE-SPECIFICBelow the volume where a batch extract strains its window, the freshness axis is the only one that differs materially and the operational axis dominates the decision. Above it, source load and partition parallelism start deciding, and the answer can flip for reasons unrelated to what any consumer asked for.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the delivery-semantics comparison underneath this one — what at-least-once costs, what makes effectively-once achievable, and why neither cadence changes those answers.
- — DevOps / Production Engineering owns the operational-burden axis in detail: what it takes to be on call for a stateful consumer versus a scheduled job, and how each is deployed and rolled back.