Flink Concepts
A stream-first distributed processor: a dataflow graph deployed once, records flowing through stateful operators, with checkpoints instead of re-runs. Compared with batch and micro-batch on what each one makes easy — not on which is better.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What changes when the job is a long-running dataflow instead of a scheduled batch, and which problems does that make easier or harder?
Consumers who need continuously updated results — an operational dashboard, a fraud signal, a feature served to a model — and who will otherwise be handed a batch table that is correct and an hour old (Stream Processing).
The unit is the record flowing through an operator, together with the keyed state that operator holds for it. Batch reasons in partitions of a bounded dataset; a stream-first engine reasons in records and the state each key accumulates over time (Streaming State).
Treat a stream processor as a batch engine with a smaller interval — same transformations, run more often. It is a serviceable first approximation and it hides the two things that actually differ: where state lives and what recovery means.
The job needs to know something about a key that happened an hour ago. In batch that is a join against a table; in a stream it is state that must be kept, sized, checkpointed and eventually expired (Stateful Stream Processing).
- The job needs to know something about a key that happened an hour ago. In batch that is a join against a table; in a stream it is state that must be kept, sized, checkpointed and eventually expired (Stateful Stream Processing).
- An event arrives late, after the window it belongs to has already emitted. Batch re-runs the day and gets it right; a stream must decide in advance how long to wait and what to do afterwards (Late Events).
- A failure means something completely different. A batch job re-runs from its input; a streaming job must restore state and rewind its source position to a consistent point, or it double-counts (Checkpointing).
- The transformation logic changes. Deploying new batch code affects the next run; deploying new streaming code requires migrating the state the old version accumulated, or starting from nothing (Schema Evolution).
- Someone treats the streaming output as reproducible. Re-running a batch job over the same input gives the same answer; a stream that depends on arrival timing and processing-time triggers may not (Determinism: Same Input, Same Output?).
What is actually happening
- The job is a dataflow graph of operators, deployed once and left running. Records flow through it continuously; there are no stages, no barriers between them and no per-run scheduling (DAG (Directed Acyclic Graph)).
- A
keyBypartitions the stream by key across parallel operator instances, which is the streaming counterpart of a shuffle — except that it happens continuously rather than as a materialised boundary (The Shuffle). - Operators hold keyed state locally: counters, aggregates, buffered joins, window contents. That state is the job's memory, and it is what makes stream processing more than a per-record function (Streaming State).
- Checkpoints are consistent snapshots of that state taken while the job runs. Markers flow through the graph in-band; each operator snapshots its state when the marker passes, so the resulting set of snapshots corresponds to one consistent cut across all operators, together with the source positions that produced it (Checkpointing).
- On failure, the job restores the last checkpoint and rewinds its sources to the positions recorded in it. Input consumption and state update are then effectively-once relative to that checkpoint. The output write is only once if the sink is transactional or the write is idempotent — that is a separate condition and it must be named (Exactly-Once: Input Consumption, State Update, Output Write).
- Event time and watermarks decide when a window can be considered complete, independently of when records actually arrived. This is the machinery that makes results depend on when things happened rather than on when they were processed (Watermarks).
A graph that is deployed, not a job that is scheduled
The structural difference is worth stating plainly before any comparison. A batch job is scheduled, plans itself, acquires resources, runs to completion and exits. A stream-first job is deployed once and stays up: its operators are placed, connected and left running, and records flow through them for months.
Everything else follows from that. There is no barrier between stages, because there are no stages — there is backpressure, which propagates upstream when an operator cannot keep up. There is no re-run, because there is no run — there are checkpoints, and recovery means restoring one and rewinding the sources to match (Backpressure).
And state is local and long-lived rather than reconstructed each time. An operator keyed by customer holds that customer's running aggregate in its own memory or state backend, which is what makes per-key computation cheap and what makes the job something you have to maintain rather than merely re-run (Streaming State).
- The barrier is in-band with the records, so every operator snapshots at the same logical point and the resulting set of snapshots is a consistent cut (Checkpointing).
- Source positions are part of the snapshot, which is what makes restore-and-replay produce effectively-once state updates rather than duplicates (Offsets and Commits).
- The sink is the part that is not covered for free. Once-only output needs a transaction that commits with the checkpoint, or an idempotent write (Exactly-Once: Input Consumption, State Update, Output Write).
Continuous, micro-batch and batch, compared on what they make easy
The honest comparison is not about which is faster. It is about which problems each model makes easy and which it makes your responsibility, and the answer depends on the latency the consumer genuinely needs and on how much state the computation carries.
Batch is easy to reason about and easy to reprocess: the input is bounded, the run is reproducible, and fixing a year of history is the same operation as computing a day of it. It is bad at latency, and no tuning removes the interval.
Continuous processing is good at latency and at computations that carry state across a long window, and it makes deployment, state migration and reprocessing into ongoing engineering work. Micro-batch sits between the two with recovery that is a re-run of a small batch, which is genuinely simpler than restoring a snapshot, at the cost of a latency floor set by the interval.
Note what does not appear in the table: a winner. A team with an hourly reporting requirement and no state should not be running a continuous dataflow, and a fraud signal that must respond within seconds cannot be a nightly batch. The requirement decides (Batch vs Streaming Ingestion).
| Dimension | Continuous dataflow (Flink-style) | Micro-batch (Spark-style structured streaming) | Scheduled batch |
|---|---|---|---|
| Unit of work | A record flowing through a deployed operator graph | A small bounded batch, planned and executed per interval | One bounded dataset per scheduled run |
| Latency floor | Per-record, bounded by watermark policy rather than by a schedule | The batch interval | The schedule interval plus the run duration |
| State | Keyed, local, long-lived, checkpointed | Explicit between batches, committed with each batch | Reconstructed from the input each run — there is no live state |
| Recovery | Restore last checkpoint, rewind sources, continue | Re-run the failed micro-batch from its recorded offsets | Re-run the job from its inputs |
| Reprocessing history | Run the same logic over bounded input, or a separate batch path | Re-run over the historical range | The native case — this is what batch is |
| Deploying a logic change | Savepoint, migrate state, restart from it | Next batch picks up the new code; state schema still needs care | Next run uses the new code |
| Cost shape | Continuous resource hold, sized for peak | Per-interval execution; overhead per batch | Per-run execution; nothing between runs |
| Operational burden | Highest: always-on, stateful, needs upgrade procedures | Middle: familiar batch tooling with streaming semantics | Lowest: a job that starts, ends and can simply be re-run |
What "exactly-once" is allowed to mean here
This is the phrase that does the most damage in streaming, and a stream-first engine is where it is most often misapplied. The claim has to be split into three separate questions before it means anything (Exactly-Once: Input Consumption, State Update, Output Write).
Input consumption: after a failure, are records re-read? Yes — the job rewinds to the positions in the last checkpoint, so records after that point are consumed again. State update: does that re-consumption double-count in the operator state? No, because the state was snapshotted at the same logical point as the positions, so restoring both together makes the effect of each record appear once. That is what "effectively-once" names, and it is a real and strong property.
Output write: does the downstream system see each result once? Only if the sink participates. A transactional sink that commits with the checkpoint gives once-only visibility; an idempotent write keyed on something stable makes a repeat harmless; a plain append to a file or an HTTP call to an external API does neither, and the same output will appear twice after a restore.
So the accurate sentence is long, and worth writing out in a design document: *effectively-once state updates under checkpointing with replayable sources, plus once-only output visibility through a transactional sink*. Anything shorter is a claim about a system, and this is a claim about three of them (Atomic Publish).
- 1Input consumption
Reads from a replayable source whose position can be rewound.
guarantees At-least-once by itself: after a restore, records after the checkpoint position are read again.
fails by A source that cannot be rewound — an HTTP push, a queue that has already acknowledged — which makes the whole chain unrecoverable (Replay from the Log).
- 2State update
Applies each record to keyed operator state.
guarantees Effectively-once, because state and source positions are snapshotted as one consistent cut (Checkpointing).
fails by A checkpoint that cannot complete — under backpressure, or with state too large — leaving the recovery point further and further behind.
- 3Output write
Emits results to a sink.
guarantees Nothing on its own. Once-only visibility requires a transactional commit tied to the checkpoint, or an idempotent write.
fails by A plain append or an external call, which repeats after a restore and duplicates downstream (Deduplication).
- 4Downstream consumer
Reads the sink.
guarantees Whatever the sink gave it, and no more. A consumer cannot recover a guarantee the pipeline did not provide (Upserts and Merges).
fails by Assuming the phrase "exactly-once" in an architecture diagram applies to the number it is reading.
Read this table before writing "exactly-once" anywhere. Two of the four rows are guaranteed by the engine; the other two are engineering you have to do.
Checkpoint mechanics, state backends, savepoint compatibility across versions, and which connectors support transactional commits all change between releases and between connectors. The model above — barriers, consistent snapshots, source rewind, sink participation — has been stable for years; the specific guarantees of any specific connector must be checked in the documentation for the version you deploy.
How to build it
Most important first.
- Choose by requirement, not by architecture fashion. What latency does the consumer genuinely need, how much state does the computation require, and how often will the logic change (Batch vs Streaming Ingestion)?
- Size and bound the state deliberately. State that grows per key without expiry is a slow-motion outage, and it is the most common way streaming jobs fail in their second year (Streaming State).
- Decide the lateness policy explicitly, with the consumer. How long a window waits, and what happens to what arrives afterwards, is a product decision expressed as configuration (Late Events).
- Name what "exactly-once" means for this job in writing: effectively-once state updates under checkpointing, plus whatever the sink provides for the output write (Exactly-Once: Input Consumption, State Update, Output Write).
- Keep a batch path for reprocessing history. A stream-first engine can run bounded input, and a stream that cannot be replayed over a corrected year is a stream you cannot fix (Replay from the Log).
- Plan state migration before the first deploy. Savepoints and state schema evolution are what make a long-running job maintainable, and they are much harder to add later (Rolling Back Data).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Checkpointing gives effectively-once state updates: after recovery, each input record has affected the operator state as if processed once, because the state and the source positions were snapshotted together (Checkpointing).
- It does not by itself give exactly-once output. That requires a transactional sink that commits with the checkpoint, or an idempotent write keyed so that a repeat is harmless (Exactly-Once: Input Consumption, State Update, Output Write).
- Ordering is per key within the dataflow, not global. Two keys have no defined order relative to each other, which is the same guarantee a partitioned log gives (Topics and Partitions).
- Windows guarantee completeness only up to the watermark and the allowed lateness. Anything later is dropped or side-routed, by your configuration and not by chance (Watermarks).
- Availability is not guaranteed by the model. A long-running job that fails restores from a checkpoint and reprocesses from those positions, so recovery has a duration and the output has a gap.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Reconcile the streaming result against a batch recomputation of the same period from the retained log. This is the only check that can tell you whether the continuously maintained number matches what the data actually says (Reconciliation).
- Monitor watermark progression and the count of records dropped as late. A watermark that stops advancing is the streaming equivalent of a stalled pipeline and produces no error at all (Freshness Monitoring).
- Both miss state that is subtly wrong from a logic change deployed mid-stream, where old state was computed one way and new records another. That divergence reconciles against nothing (Semantic Changes).
- A continuous dataflow removes the batch interval from end-to-end latency: results update as records arrive rather than when a schedule fires (Cost vs Freshness).
- It does not remove waiting. Event-time windows deliberately wait for the watermark, so a five-minute window with generous lateness is not fresher than a five-minute batch — it is more correct about which records belong where (Event Time).
- Freshness under a stream is continuous but not free of gaps: a restore from checkpoint reprocesses from recorded positions, and the output lags during that recovery (Pipeline Reliability).
- The transformation and its state evolve together. Changing an aggregate's shape means migrating the state that already holds the old shape, which is a data migration performed on a running system (Schema Evolution).
- A schema change in the source stream reaches a running job immediately rather than at the next scheduled run, so contract enforcement at the boundary matters more here than in batch (Contract Enforcement).
- Restarting from empty state is always available and is rarely acceptable, because the state encodes history that the input may no longer contain within its retention (Retention and Replay).
- Recovery is restore-and-replay: reload state from the last checkpoint, rewind sources to the positions it recorded, and continue. It is automatic and its cost is the reprocessing gap (Checkpointing).
- A savepoint is the same mechanism used deliberately — for an upgrade, a scaling change or a configuration change — and it is what makes a long-running job maintainable rather than immortal (Rolling Back Data).
- Fixing history is a batch problem even in a stream-first world: reprocess the corrected range from the retained log and republish, rather than trying to rewind a live job into the past (Planning a Backfill).
What can go wrong
- Unbounded state growth from keys that never expire, ending in an out-of-memory failure months after deployment (Memory Pressure, Swap and the OOM Killer).
- A watermark that stalls because one source partition is idle, holding every window open and emitting nothing while the job stays perfectly healthy (Watermarks).
- Checkpoints that take longer than the interval between them, so the job spends its time snapshotting rather than processing (Pipeline Observability).
- A non-transactional sink turning effectively-once state into duplicated output after a restore (Deduplication).
- The mitigation failing: adding allowed lateness to catch late events, which holds every window open longer, grows state, and delays every result for the sake of a small minority of records (Late Events).
- "Streaming is more modern, therefore better." They answer different questions. Compare freshness requirement, state size, reprocessing needs, operational burden and team familiarity, then choose — and note that most analytical questions do not need sub-minute answers (Batch vs Streaming Ingestion).
- "Flink gives exactly-once." Checkpointing gives effectively-once *state updates* with source rewind. Output is once only with a transactional or idempotent sink, and that has to be built (Exactly-Once: Input Consumption, State Update, Output Write).
- "A stream can replace the batch pipeline entirely." Until you need to reprocess a corrected year. Then you need bounded processing over history, which is batch by another name (Kappa Architecture).
- "Micro-batch is just streaming with worse latency." It also has different recovery, different state handling and different operational properties. The latency is the most visible difference and not the most important one.
Operating it
- Watermark lag per operator, which is the stream-first equivalent of freshness and the first thing to look at in any incident (Freshness Monitoring).
- Checkpoint duration, size and failure count. A growing checkpoint is growing state, and growing state is the failure that arrives slowly (Pipeline Metrics).
- Backpressure per operator, which localises the bottleneck to a specific stage of the dataflow in a way batch stage metrics cannot (Backpressure).
- Records dropped as late, by window, which is the number the lateness policy is actually producing rather than the one it was assumed to produce.
- At 10x throughput, parallelism per operator increases and the model is unchanged — this is the well-behaved axis.
- At 10x key cardinality, state grows proportionally and becomes the constraint. Cardinality is the dimension that decides whether a streaming design is viable (Partition Cardinality).
- At high parallelism, checkpoint coordination and the size of the snapshot become significant, and checkpoint interval turns into a real tuning decision rather than a default.
- A long-running job holds resources continuously, whether or not data is flowing. Its cost is a function of uptime rather than of volume, which is the opposite of a batch job and can be cheaper or far more expensive depending on the duty cycle (Compute Waste).
- State storage and checkpointing to durable storage are ongoing costs that grow with key cardinality rather than with throughput.
- Operational cost is real and frequently understated: a job that must run continuously needs upgrade procedures, state migration and on-call attention that a nightly batch does not (Scoring Operational Complexity).
- Continuous processing buys latency and costs an always-on system with state to manage, migrate and monitor.
- Batch buys reproducibility, easy reprocessing and simple deployment, and costs an interval of latency that no amount of tuning removes (Batch vs Streaming Ingestion).
- Micro-batch sits between them and is a genuine middle rather than a compromise: recovery is a re-run of a small batch, latency is bounded by the interval, and state between batches is explicit (Batch and Streaming Unification).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ENGINE-SPECIFICBarrier-based checkpointing of a continuously deployed dataflow is Flink's architecture; Spark Structured Streaming executes micro-batches with offsets and state committed per batch; Kafka Streams runs as a library inside your own application with state in local stores backed by changelog topics. All three do event time and state, and their recovery models differ in ways that matter operationally.
- GENERALContinuous dataflow with keyed state, event time and periodic consistent snapshots is a model, not a product. The concepts here — watermark, keyed state, checkpoint, savepoint, backpressure — appear under those or similar names in every serious stream processor.
- SIMPLIFIEDThe description leaves out state backends and their very different performance profiles, incremental checkpointing, unaligned checkpoints under backpressure, and the details of rescaling a job with existing state. Each of those matters in production and none of them changes the model a learner needs first.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the consistent-snapshot algorithm that barrier-based checkpointing implements, and the general result that a consistent global state can be recorded without stopping the system.
- — DevOps / Production Engineering owns deploying a change to an always-on stateful job: savepoint, migrate, restart, and the rollback path when the new version is wrong.