Streaming Ingestion
Event happens, producer publishes, broker durably holds, consumer reads, storage lands it. Continuous rather than windowed — and continuously running.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What does moving ingestion from a schedule to a continuous consumer actually change about what can be lost, duplicated, reordered or delayed?
Two very different ones, and conflating them causes most streaming platform confusion. A stream consumer wants low-latency events and tolerates re-reading. An analytical consumer wants files in object storage that are large, deduplicated and partitioned for scanning. Streaming ingestion serves the second by continuously producing what the first would call an artefact (The Raw Landing Zone).
One event, as the producer chose to model it. That choice is upstream of everything: an event may be a fact about the world ("order placed"), a row change ("orders.status 2 → 5"), or a snapshot of an entity. Only the first survives a change to the source's schema with its meaning intact, and only the second reconstructs state without ambiguity (What a CDC Event Contains).
Point a consumer at the topic, write each message to object storage as it arrives, commit the offset. It is a short program, it has no schedule, and the data is in the lake within seconds of the event happening.
One file per message. After a day there are millions of tiny objects, and every query downstream spends its time listing and opening files rather than reading data (File Size and the Small-Files Problem).
- One file per message. After a day there are millions of tiny objects, and every query downstream spends its time listing and opening files rather than reading data (File Size and the Small-Files Problem).
- The offset is committed before the write is durable. A crash between the two loses every message in flight, permanently, and the consumer restarts from a position that says the work was done (Offsets and Commits).
- The offset is committed after the write, so a crash re-delivers. The same event is now in the lake twice and every downstream aggregate is high until something deduplicates (Deduplication).
- The consumer falls behind. Nothing errors — lag simply grows — until it crosses the broker's retention and messages are deleted before being read. That transition is silent and the data is unrecoverable (Retention and Replay).
- A producer bug publishes malformed events. The consumer throws, restarts, throws again, and either stops the whole partition or skips the message, depending on a configuration decision nobody remembers making (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Events arrive out of order across partitions, and a downstream "latest state per key" computed by arrival produces a stale value that looks entirely current (CDC Ordering and Transaction Boundaries). A partition count change makes this worse rather than better: keys are re-mapped, one entity's history is split across two partitions, and the ordering guarantee — which was only ever per-partition — no longer covers that entity at all (Event Keys and Partition Assignment).
What is actually happening
- Streaming ingestion replaces a schedule with a position. There is no window; there is an offset per partition, and progress is the act of moving it. Everything that batch expresses as "which windows are complete" becomes "how far behind is each partition".
- The broker turns an ephemeral event into a durable, replayable, ordered-within-partition record. That is the actual product: not speed, but the ability for many independent consumers to read the same history at their own pace and to re-read it (Kafka as a Log, Not a Queue).
- Delivery is at-least-once end to end unless every link is idempotent. Producers retry on unacknowledged sends, brokers may accept a duplicate, and consumers re-read after a crash. Each of those is individually correct behaviour and they compose into duplicates (At-Least-Once Delivery).
- The consumer is a long-lived stateful process, and that is the largest operational difference from batch. It has a position to commit, membership in a group to maintain, a rebalance to survive, and a lag that must be watched — none of which exists in a job that runs and exits (Consumer Groups and the Parallelism Ceiling).
- Landing a stream in analytical storage requires buffering against two conditions: enough time or enough bytes. That buffer is where a continuous stream is converted back into discrete files, and its size is the single decision that determines whether the lake is queryable (File Compaction).
- Event time and arrival time diverge continuously rather than at window boundaries. Streaming does not remove the late-data problem; it makes lateness a spectrum handled by watermarks instead of a boundary condition handled by a lookback (Watermarks).
Five hops, and the one that decides whether you lose data
The streaming path has more moving parts than the batch one and only one of the seams is genuinely dangerous. Everything upstream of the broker can be retried; everything downstream of a durable write can be recomputed. The seam that decides correctness is the ordering of two operations at the consumer: writing the data, and committing the position.
Commit the position first and a crash between the two loses every message in flight, permanently, with the consumer restarting from a position that asserts the work was done. Commit it second and a crash re-delivers, which is a duplicate — bounded, detectable and fixable by a keyed write. The choice is between a silent unrecoverable failure and a noisy recoverable one, and it is one line of code.
The other seam worth naming is before the broker entirely. A producer that commits a database transaction and then fails to publish has created an event that exists in the source and nowhere else, and no consumer-side check can distinguish that from an event that never happened. Solving it requires the write and the publish to share a transaction, which is what the outbox pattern does (The Transactional Outbox).
- 1Producer
Publishes an event after the state change it describes has been committed.
guarantees Only that events it successfully published were published. Says nothing about events it intended to publish and did not.
fails by Committing to the database and crashing before publishing — the dual-write problem, invisible from every downstream vantage point (The Dual Write Problem).
- 2Broker
Appends the event to a partition and replicates it.
guarantees Durable within its acknowledgement settings, ordered within a partition, replayable within retention. No cross-partition ordering, ever.
fails by Expiring data at retention while a consumer is still behind it — the only failure here that cannot be recovered by catching up.
- 3Consumer
Reads forward from its committed offset, decodes, and hands events to the writer.
guarantees That it will process everything from its offset onward, at least once.
fails by Rebalances that pause processing; poison messages that block a partition; falling behind without erroring.
- 4Buffer and write
Accumulates events until a size or time threshold, then writes one file to object storage.
guarantees Durability once the write completes, and file sizes suited to analytical reads if the threshold was chosen for that.
fails by Buffering in memory and losing the buffer on crash; or writing per message and destroying downstream query performance.
- 5Commit the offset
Records how far the consumer has durably landed data.
guarantees A correct restart position — but only if this happens strictly after the write is durable.
fails by Being done first, which converts every crash into silent permanent loss of the in-flight batch (Offsets and Commits).
Compare with the batch pipeline: the shape is the same — bound, read, move, commit data, commit position — and the same rule decides correctness. Streaming changes the cadence and the failure surface, not the underlying obligation.
The failures that do not raise an exception
Streaming platforms are better instrumented than most batch pipelines, and their characteristic failures still go unnoticed, because the important ones present as gradual changes in a number rather than as errors. Lag grows. File count grows. A dead-letter queue fills. Nothing throws.
The retention cliff deserves particular attention because it is the only failure in this module where a purely quantitative change becomes a qualitative one at an invisible threshold. A consumer forty minutes behind and a consumer forty-nine hours behind look identical on a lag chart with a linear axis; if retention is forty-eight hours, one of them has lost data and the other has not.
This is why lag alerting must be framed against retention rather than against a round number. "Alert at one hour" is a number somebody liked. "Alert at one quarter of retention" is a statement about how much time the on-call engineer has before the failure becomes permanent.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Consumer throughput drops below production rate | Lag climbs steadily. No errors. Dashboards get gradually staler. | Slower processing per message, fewer consumer instances, or a partition rebalance that left work unclaimed. | Scale consumers up to but not beyond the partition count, and if that ceiling is the constraint, the partition count is the thing to change (Topics and Partitions). |
| Lag exceeds broker retention | Consumer resumes and simply starts from the oldest available offset. Data between its last position and that point is gone. | The broker deleted segments the consumer had not read. Working as configured. | Alert on lag as a fraction of retention, not as an absolute. After the fact, the only repair is whatever the source system can still be re-extracted for (Ingestion Failure & Recovery). |
| A message fails to decode | Either one partition stops entirely, or messages vanish — depending on a configuration flag. | No dead-letter path, so the only options are block or skip, and both are lossy in different directions. | Route to a dead-letter destination with raw bytes and offset, alert on any arrival, and keep the partition flowing (A Dead-Letter Queue Is a Workflow, Not a Bin). |
| Consumer restarts after a crash | A block of events appears twice in the lake. Counts are high for that period. | At-least-once redelivery from the last committed offset — correct behaviour, not a bug. | Deduplicate on the producer event id at the staging layer. Do not attempt to fix this in the consumer; the consumer cannot know what landed (Deduplication). |
| Partition count increased | One entity's events stop being ordered relative to its own history at the moment of the change. | Key-to-partition mapping is a function of partition count, so existing history stays where it was and new events go elsewhere. | Treat it as a breaking change with a cutover, and make downstream state reconstruction order by a source-provided version rather than by offset (Event Keys and Partition Assignment). |
| Producer crashes between database commit and publish | The source has a row nothing downstream ever saw. Reconciliation against the source finds it; nothing else can. | Two systems written to without a shared transaction. | Publish through an outbox written in the same transaction as the state change, and relay from there (The Transactional Outbox). |
Turning a continuous stream back into files worth querying
The last hop of streaming ingestion is the one most often treated as an implementation detail, and it is where the analytical consumer's experience is decided. A stream is continuous; object storage and columnar formats want large, immutable, well-organised files. Something has to convert between them and that something is a buffer.
The conversion has two parameters: how much to accumulate before writing, and how to lay the resulting files out. Get the first wrong and you produce either a small-file problem or a freshness problem. Get the second wrong — most commonly by partitioning on event time so that late events must rewrite old partitions — and every re-run becomes unbounded.
Partition by arrival. It is the property ingestion actually knows, it makes each partition immutable once its interval passes, and it bounds any re-run to the intervals it touched. Event-time organisation is a modelling concern that belongs to the transformation, which can read arrival partitions and write event-time ones with full knowledge of lateness (Late-Arriving Data).
The layout below shows the same day of data organised both ways, under a query for a single day of events. Notice that the arrival layout reads more partitions than strictly necessary and never has to rewrite one; the event-time layout reads exactly one and has to mutate it every time a late event shows up.
- arrival/ingest_date=2026-03-14/hour=00..23/a day of arrivals, whatever their event date · 24 files · read
- arrival/ingest_date=2026-03-15/hour=00..23/a day of arrivals, including stragglers from the 14th · 24 files · read
- arrival/ingest_date=2026-03-16/hour=00..23/a day of arrivals, including the tail of the 14th · 24 files · read
- arrival/ingest_date=2026-03-13/hour=00..23/a day of arrivals, all with earlier event dates · 24 files · skipped
- event/event_date=2026-03-14/exactly the target day · 1 file · read
- event/event_date=2026-03-13/the previous day · 1 file · skipped
The counts are illustrative structure, not measurements. The teaching is the asymmetry: arrival partitions are append-only and bounded to re-run; event-time partitions are ideal to query and must be mutated as lateness arrives. Land by arrival, model by event time.
Inversely proportional to buffer size and directly proportional to partition count. The largest downstream cost and the one incurred by a decision made at ingestion.
Pure rework: reading and rewriting data that was written at the wrong size. Entirely avoidable by buffering correctly the first time (File Compaction).
Equal to the buffer interval, exactly. The one cost on this list that the analytical consumer feels directly and can therefore be asked about.
Grows with buffer size and with partition count together; the reason buffers are bounded by bytes as well as by time.
Independent of buffering, but it is the budget that decides how long a landing failure can last before it becomes permanent.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a lake-landing consumer, given to establish ordering. The point is that a buffer is not a performance tuning knob — it is a trade between one cost the analytical consumer sees immediately (freshness) and one they will pay every day thereafter (file count).
How to build it
Most important first.
- Commit the offset after the write is durable, always. This makes the pipeline at-least-once, which is a bounded, detectable, fixable problem — and makes deduplication a downstream requirement rather than a hope. Deduplicate on a producer-assigned event id, never on arrival order or on a hash of the payload — payload hashing merges genuinely distinct events that happen to be identical (Idempotent Data Pipelines, Duplicate Rows).
- Buffer to a size-or-time threshold before writing, and choose the size against what the read format wants rather than against the desire for freshness. A thirty-second buffer that produces tiny files is a freshness gain paid for by every future query (Parquet).
- Partition landed data by arrival, not by event time. Arrival partitioning makes a re-run bounded and re-playable; event-time partitioning means a late event must rewrite an old partition, which is a mutation of history at ingestion time (Partitioning).
- Key events by the entity whose ordering matters, so per-partition ordering is per-entity ordering. Ordering you did not key for does not exist (Event Keys and Partition Assignment).
- Route unparseable messages to a dead-letter destination with the raw bytes and the offset, rather than skipping or blocking. Both alternatives lose information: skipping loses the event, blocking loses everything behind it (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Alert on consumer lag in *time units* and set the threshold well inside the broker's retention, because the retention boundary is where recoverable becomes unrecoverable (Depth Is Not an Emergency; Age Is).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Delivery: at-least-once from producer to landed file, in any design that retries. Getting to effectively-once requires naming which of three things you mean — the consumer's offset commit, any state the consumer holds, and the write to storage — and buying each separately. A transactional sink plus offset commit in the same transaction gives it for the last two; nothing gives it for a producer that retried before its first send was acknowledged (Exactly-Once: Input Consumption, State Update, Output Write).
- Ordering: per partition only. Across partitions there is none, and there never was — a claim that a topic is ordered is a claim about a one-partition topic (Topics and Partitions).
- Completeness: bounded by what producers actually published. The broker cannot know about an event that a crashed producer never sent, and no consumer-side check can distinguish that from an event that never happened (The Dual Write Problem).
- Durability: as strong as the broker's replication and acknowledgement settings, and then only until retention expires. Durability with a deadline is a different property from durability.
- Freshness: bounded by consumer lag plus the landing buffer. Both are continuous quantities that degrade gradually, which is a genuinely different failure shape from a batch window that is either there or not.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Offset continuity per partition: assert that landed data covers every offset from the last committed position with no gaps. It is the streaming equivalent of window continuity and it catches a skipped message, a partition nobody consumed, and a rebalance that lost a claim.
- Uniqueness on the producer event id across a landing window, which measures how much duplication the at-least-once path is actually producing rather than assuming it is negligible.
- A count of events published versus events landed, per topic per period, requires the producer to emit a count — which is worth asking for. Without it, completeness against the *world* is unmeasurable and only completeness against the broker can be checked (Reconciliation).
- What none of this sees: an event the producer never published because it crashed after committing its own transaction. That gap is invisible from the broker onward and is the reason the transactional outbox pattern exists (The Transactional Outbox).
- The shape is continuous: staleness equals lag, and lag responds to load rather than to a clock. Under normal conditions this is the freshest ingestion available; under load it degrades smoothly and without an obvious threshold.
- The landing buffer sets a floor. Writing files every few minutes means analytical consumers see data on that cadence regardless of how fast the consumer reads, so "streaming ingestion" and "streaming freshness for analysts" are different claims (Cost vs Freshness).
- Lag is not evenly distributed. One hot partition can be far behind while the aggregate looks healthy, so per-partition lag is the signal and an average is a way of hiding the problem (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- The failure that matters is not slow, it is silent: lag that grows past retention. Freshness monitoring for a stream must be framed against retention, because that is where a latency problem becomes a data-loss problem.
- Producers and consumers deploy independently, so a schema change is live in production with both versions running simultaneously. A schema registry with compatibility rules is not ceremony here; it is the only thing preventing a producer from breaking every consumer at deploy time (Schema Registry).
- Landing raw messages in a self-describing format preserves the ability to reprocess events written under an older schema. Landing them as bare JSON with an assumed shape does not, and the loss appears only when you reprocess (Avro).
- Changing partition count re-maps keys to partitions. Existing history stays where it was, so one entity now has events in two partitions and its per-partition ordering guarantee no longer covers its own history (Event Keys and Partition Assignment).
- Changing what an event *means* — a
statusvalue repurposed, an amount switching from gross to net — passes every compatibility check the registry performs, because none of them are about meaning (Semantic Changes).
- Recovery is replay from an offset, and it is the strongest recovery mechanism in this module — provided the offset is still within retention. Reprocessing a week of history is a configuration change rather than an engineering project (Replay from the Log).
- Retention is therefore a recovery-window decision, not a storage-cost decision. The right question is "how long can this consumer be broken before data is lost", and the answer should be argued with the people who would be paged (Retention and Replay).
- Replaying into a sink that appends produces duplicates for the entire replayed range. Replay is only safe where the write is keyed and idempotent, which is the same requirement as everywhere else in this module but is exercised far more often here.
- Once retention has passed, the stream is not a recovery path at all and the only remaining source is whatever landed in raw storage — which is why landing raw untouched matters more for streaming than for batch, not less (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- Offset committed before the write is durable, turning every crash into silent permanent loss.
- Consumer lag crossing broker retention. Recoverable right up until it is not, with no discrete event marking the transition.
- A rebalance storm: a slow consumer misses its heartbeat, the group rebalances, processing pauses, the next consumer is also slow, and the group spends its time rebalancing instead of consuming (Consumer Groups and the Parallelism Ceiling).
- A poison message blocking a partition indefinitely, so one malformed event stops all data behind it while every other partition looks healthy — or the mitigation failing in the other direction, where a dead-letter queue absorbs bad messages, nobody consumes it, and a growing fraction of the data is parked behind a green pipeline.
- Millions of tiny files in the lake — ingestion is perfectly healthy and every downstream query gets slowly worse (File Size and the Small-Files Problem).
- A producer that commits its database transaction and then fails to publish. Nothing downstream can detect it, because the event simply never existed as far as the broker is concerned (The Dual Write Problem).
- "Streaming means real-time." It means continuous. End-to-end freshness for an analytical consumer is consumer lag plus landing buffer plus whatever the transformation schedule is, and a streaming ingest feeding an hourly model gives hourly data (Batch vs Streaming Ingestion).
- "The broker guarantees exactly-once." Scope it: to input consumption, to state update, or to output write? There are configurations giving effectively-once between a broker and a consumer's state and output when both commit together. None of that covers a producer that retried a send whose acknowledgement was lost, and none of it covers a duplicate the source system itself generated (Exactly-Once: Input Consumption, State Update, Output Write).
- "Lag is a performance problem." Lag is a performance problem until it approaches retention, at which point it is a data-loss problem. Those need different alerts and different responses.
- "We do not need a raw landing zone, we have the topic." Retention is a deadline. The topic is a recovery path with an expiry date, and after it passes the only copy is whatever you landed (Keeping Raw History: The Recovery Position and the Liability).
- "Ordering is guaranteed." Per partition. Any statement about ordering that does not contain the word "partition" is either about a single-partition topic or is wrong (CDC Ordering and Transaction Boundaries).
- A topic is a durable copy of whatever the producer put in it, replicated across brokers and retained for a fixed window. If events carry personal data, deleting it means waiting out retention or rewriting a compacted topic — neither of which is a targeted delete (Deletion Requests).
- Many independent consumers can read a topic, and the producer generally cannot see who. Access control on topics is therefore the real boundary for event data, and it is frequently coarser than the equivalent control on the source database (Data Access Control).
Operating it
- Consumer lag per partition, expressed in time, with the broker's retention drawn on the same axis. This is the single most important chart in a streaming ingestion platform (Depth Is Not an Emergency; Age Is).
- Landed bytes and file count per interval. The ratio between them is average file size, which is the early warning for a small-file problem that will otherwise show up months later as query cost (Scan Cost).
- Dead-letter arrival rate, alerted on any non-zero value rather than on a threshold. A dead-letter queue that nobody alerts on is a data-loss mechanism with good intentions.
- Duplicate rate measured after landing: distinct event ids over total events, per window. It quantifies the at-least-once path instead of assuming it is negligible (Duplicate Rows).
- Rebalance frequency per consumer group, because frequent rebalances present as unexplained lag rather than as errors (Twenty Workers, All Busy, Five Hundred Waiting).
- At 10x event volume, partition count becomes the parallelism ceiling: a consumer group cannot have more active consumers than partitions, so throughput is capped by a decision made when the topic was created (Topics and Partitions).
- At 100x, key skew dominates. One hot key pins one partition to one consumer, and that consumer's lag decides the platform's freshness regardless of how much capacity the others have (Data Skew).
- Topic count scales the operational surface: each topic is a schema, a retention policy, a partition count, a consumer group and a lag alert. Fifty topics is a platform; five hundred is a full-time job (Data Platform Engineering).
- What scales well is replay. Reprocessing a large history is bounded by consumer throughput and costs nothing at the source, which is the opposite of batch backfills that re-load the origin system.
- Continuous consumer compute, paid whether or not events are flowing. For a low-volume topic this is the dominant cost and it is entirely unlike batch, where idle costs nothing (Compute Waste).
- Broker storage for the retention window, which is a direct multiplication of event volume by how long you keep it — and that duration is a recovery decision being paid for in storage.
- File count downstream, driven by buffer size and partition count together. Short buffers across many partitions multiply into a small-file problem faster than either factor suggests alone (File Compaction).
- Compaction compute to fix the file count afterwards, which is real work that exists only because the landing step wrote files at the wrong size (What Actually Drives Data Platform Cost).
- Streaming ingestion buys continuous freshness, independent multi-consumer reads and replay-based recovery. It costs a permanently running stateful component, a retention window that bounds recovery, and a small-file problem that must be actively managed rather than avoided.
- The operational burden is the honest cost and it is not small: lag alerting, rebalance behaviour, poison-message handling, schema compatibility and partition sizing are all live concerns that a scheduled extract simply does not have (Batch vs Streaming Ingestion).
- Larger landing buffers buy fewer, larger, cheaper-to-query files and cost freshness in exactly the amount buffered. This is the clearest dial in the design and it should be set by the analytical consumer, not by the streaming one.
- Keying by entity buys per-entity ordering and costs partition balance, because real entity distributions are skewed and the key that gives you the ordering you want is often the one that gives you a hot partition (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka orders per partition and retains by time or size with consumer-managed offsets; Kinesis orders per shard with a shard count you resize; Pub/Sub provides no ordering at all unless an ordering key is set and acknowledges per message rather than by offset — so "replay from a position" is not a uniform operation across them.
- GENERALThe structure — producer, durable log, positioned consumer, buffered landing — and the requirement to commit the position after the write are properties of the pattern, not of any product, and apply equally to a managed queue and a self-run cluster.
- SCALE-SPECIFICBelow the volume where a scheduled extract misses its window, streaming ingestion adds a permanently running component and a retention cliff in exchange for freshness nobody requested. It becomes the better answer when many independent consumers need the same data, or when replay-based recovery is worth more than schedule simplicity.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns consumer-group membership, rebalancing, leader election in the broker, and what an acknowledgement means when the network can drop the acknowledgement rather than the message.
- — DevOps / Production Engineering owns deploying a stateful consumer without losing its position, and draining one gracefully so in-flight events are landed before the process exits.