The question this answers
If I can never know an event-time window is complete, what makes it fire?
A watermark at time T is an assertion by the processor, not a property of the data: it claims no further records with event time before T are expected. That claim can be wrong in both directions — records may still arrive (producing Late Events: The Window Already Fired), and the watermark may lag far behind reality (delaying every window). No watermark strategy is correct in general; each trades completeness against latency.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The processor knows the maximum event time it has observed on each input partition, and how long ago it observed anything at all. It does not know whether a source is slow, idle, or dead, whether an offline client is about to reconnect with a day of data, or whether the partition it has heard nothing from is empty or broken. The watermark is an inference built entirely from local observation, which is why it is a heuristic and why saying so out loud changes how people operate it.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What a watermark is, precisely
A watermark is a monotonically advancing marker on the event-time axis, carried through the pipeline alongside the records. When the watermark reaches time T, every operator downstream may conclude that windows ending at or before T can be finalised — and that conclusion is the only reason an event-time window ever closes.
The framing that keeps people out of trouble: a watermark is a statement of belief with a deadline attached, not a measurement. "I believe I have seen everything up to 10:00, and I am willing to act on that belief now, because waiting longer costs more than being occasionally wrong." That is the same structure as a failure detector deciding a node is down: an unfalsifiable claim, made because the alternative is waiting forever.
Monotonicity is the one property that must hold. A watermark that went backwards would un-finalise already-closed windows, so implementations force it forward: a watermark never retreats, even if a record arrives that would justify retreating. That record becomes a late event instead, which is exactly how the two concepts fit together — the watermark decides when to stop waiting, and lateness policy decides what happens to whatever shows up after.
How watermarks are generated, and why each strategy is a bet
There is no canonical algorithm; each strategy encodes a different assumption about the source, and each is wrong in a way you should be able to name.
Bounded out-of-orderness is the workhorse: watermark equals the maximum event time seen minus a fixed allowance, say two minutes. It bets that records are at most that far out of order. Cheap, predictable, and wrong for anything with a long tail — the offline mobile client blows straight through it.
Percentile-based strategies track the observed skew distribution and set the allowance at, say, p99. Adaptive and better-behaved when the distribution shifts, and harder to reason about during an incident because the watermark’s position now depends on recent history.
Source-aware strategies use knowledge the data itself provides: a partner file with a manifest declaring its time range, or a source that emits an explicit end-of-period marker. This is the only approach that is not a guess, and it is available only when the source cooperates — which is worth asking for, because a producer that can say "I have sent everything through 10:00" converts a heuristic into a fact.
Perfect watermarks exist only for replays of bounded, already-complete historical data, where the processor can genuinely know nothing more is coming. That case is real and useful — it is why a replay of last month can produce exactly correct windows — and it does not generalise to live streams.
| Strategy | The bet | Fails when | Cost of being wrong |
|---|---|---|---|
| Max event time − fixed allowancetypical | Out-of-orderness is bounded by the allowance | A source goes offline and returns | Records arrive after the window closed |
| Percentile of observed skewtypical | Recent history predicts near-future skew | Distribution shifts abruptly | Watermark too fast or too slow, and it moves on its own |
| Source-declared completenessassumption | The source knows and tells the truth | The source lies or the marker is lost | Windows close on a false claim of completeness |
| Perfect (bounded replay)protocol | The dataset is complete and known | Applied to a live stream | Not applicable — correct by construction for finite input |
| Processing-time fallbacktypical | Advance on wall clock when no data arrives | A source is slow rather than idle | Windows fire without data that was on its way |
The minimum rule and the idle-partition stall
When an operator has several inputs — several partitions, or several upstream operators — its watermark is the minimum across them. It has to be: the operator can only claim completeness up to time T if *every* input has reached T, and the slowest one governs.
This produces the most common watermark incident in production. One partition goes quiet — a key range with no traffic overnight, a producer that stopped, a partition whose leader is unavailable. It never advances its watermark, so the minimum never advances, so no window anywhere in the job fires. Records keep arriving on the healthy partitions and keep accumulating in open windows. Memory grows. Output stops entirely.
The symptom set is distinctive and misleading: consumer lag is fine (records are being read), CPU is low, there are no errors, and the job emits nothing. Teams look for a crash and find a healthy process. The metric that names it directly is watermark age — wall clock minus current watermark — and it is rarely on a default dashboard.
The standard mitigation is idleness detection: if an input produces nothing for a configured period, mark it idle and exclude it from the minimum. This unsticks the job and introduces a new risk, which is worth stating plainly — an input marked idle that later produces records will produce late ones, because the watermark advanced without it. You have traded a stall for lateness, deliberately.
1// Per input partition2onRecord(p, r):3 maxEventTime[p] = max(maxEventTime[p], r.eventTime)4 lastActivity[p] = now()5 6// Operator watermark: the slowest non-idle input governs7watermark():8 active = [p for p in inputs if now() - lastActivity[p] < IDLE_TIMEOUT]9 if active is empty:10 return previousWatermark // never go backwards; never invent time11 candidate = min(maxEventTime[p] for p in active) - OUT_OF_ORDER_ALLOWANCE12 return max(previousWatermark, candidate) // MONOTONIC, always13 14// Note the two failure modes this code contains, both deliberate:15// 1. Without IDLE_TIMEOUT, one silent partition stalls every window forever.16// 2. With it, a partition that wakes up produces late records by construction.17// There is no setting that avoids both.Operating a watermark
Treat watermark age as a first-class SLI. If your allowance is two minutes, watermark age in the steady state should be a little over two minutes; when it is thirty minutes, windows are firing thirty minutes late and every downstream freshness expectation is broken. Alert on it directly rather than inferring it from output volume, which lags and is easily confused with a quiet period.
Expose the per-input watermark, not only the operator minimum. The whole diagnostic value is in seeing *which* input is holding the minimum down — that single view turns a "the job stopped emitting" mystery into a ten-second answer.
And be explicit in documentation and in code comments that the watermark is a heuristic. Teams that believe it is a guarantee interpret every late record as a bug and go looking for a fault that does not exist. Teams that understand it is an estimate ask the productive question instead: is the allowance calibrated, and is the lateness policy handling the tail we knowingly chose not to wait for?
operator: hourly-revenue watermark: 2026-08-25 09:05:00 age: 4h 12m ALERT input max event time last record state partition 0 2026-08-25 13:16:41 2s ago active partition 1 2026-08-25 13:16:38 1s ago active partition 2 2026-08-25 09:07:00 4h 11m ago active <-- HOLDING partition 3 2026-08-25 13:16:40 3s ago active open windows: 1,204,556 state size: 8.4 GB and growing windows fired (last hour): 0 Partition 2's producer stopped at 09:07. Nothing is broken; nothing can fire.
Key points
- A watermark is the processor asserting "I believe I have seen everything up to T" — an estimate, never a measurement.
- It must advance monotonically; a record that would justify going backwards becomes a late event instead.
- An operator’s watermark is the minimum across its inputs, so one idle or stalled input stalls every window in the job.
- Idleness detection unsticks the minimum and guarantees that the excluded input’s records will be late when it returns.
- Watermark age (wall clock minus watermark) is the SLI that names this whole class of failure; per-input watermarks are what diagnose it.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Each source tracks the maximum event time it has observed and when it last saw a record.
- • A per-source watermark is proposed, typically as maximum observed event time minus an out-of-orderness allowance.
- • The operator takes the minimum across non-idle inputs and clamps it to be non-decreasing.
- • The watermark is propagated downstream alongside records, so every operator inherits an estimate consistent with its inputs.
- • When the watermark passes a window’s end (plus allowed lateness, where configured), that window fires and its state is released.
- • A partition goes idle and pins the minimum, so no window fires anywhere in the job.
- • The out-of-orderness allowance is too small and a large fraction of records arrive after their windows closed.
- • The allowance is too large and every result is delayed by it, breaking freshness expectations downstream.
- • A record with a far-future event time jumps the maximum forward, advancing the watermark past valid data and making everything else late.
- • Idleness detection excludes an input that was merely slow, so its records arrive late by construction.
- • A restart loses watermark state and the job re-derives it from scratch, briefly firing or withholding windows incorrectly.
- • Silent watermark stall: the job emits nothing for hours. Consumer lag is near zero, CPU is low, there are no errors, and memory climbs steadily as open windows accumulate. The cause is one partition whose producer stopped.
- • Future-timestamp poisoning: a single device with a clock set to 2106 advances the maximum event time, the watermark jumps forward by decades, every open window fires immediately with partial data, and every subsequent real record is late.
- • Chronic lateness from a tight allowance: 4% of records arrive after their window closed and are dropped. The pipeline reports healthy; a downstream total is consistently short and the discrepancy is attributed to the source system.
- • Freshness breach from a generous allowance: an allowance raised to an hour to reduce lateness now delays every result by an hour, and an operations dashboard built on those windows becomes useless for its actual purpose.
- • Idleness whiplash: an overnight-quiet partition is marked idle, and when morning traffic starts its first records are all late and dropped. The failure recurs every day at the same time and looks like a scheduled job problem.
- • State growth during a stall: window state reaches tens of gigabytes because nothing can fire, and the job eventually OOMs — a memory incident whose root cause is a time incident.
- • The minimum rule is coordination: an operator cannot claim completeness beyond what its slowest input has established, so watermark propagation couples every input’s progress to every downstream window.
- • That coupling is why a single stalled partition has job-wide blast radius, and it is unavoidable if the completeness claim is to mean anything.
- • Idleness detection is a deliberate weakening of that coordination — excluding a participant so the rest can proceed, at the cost of correctness for the excluded one. It is the same trade as a quorum choosing to proceed without a slow replica.
- • The watermark never goes backwards, so already-finalised windows are never invalidated, whatever arrives afterwards.
- • A stalled watermark preserves correctness — nothing fires early — while sacrificing liveness entirely, and unbounded state growth is the price.
- • An over-advanced watermark sacrifices completeness: windows fire without data that was genuinely on its way, and those records become late.
- • Detect: alert on watermark age against an explicit threshold derived from your allowance, and expose per-input watermarks.
- • Contain: identify the holding input from the per-input view. Restarting the job does not help and usually makes state worse.
- • Recover: restart the stalled producer, or mark the input idle so the minimum can advance — knowing this makes its future records late.
- • Reconcile: reprocess the affected event-time range from the log to recover windows that fired with incomplete data.
- • Verify: watermark age back to roughly the allowance, windows firing at the expected rate, and window state size back to baseline.
- • Watermark age (wall clock minus current watermark) per operator — the primary SLI for this whole class of failure.
- • Per-input watermark and time since last record, which identifies the holding input immediately.
- • Windows fired per interval against windows expected, which catches a stall before memory does.
- • Records dropped or side-outputted as late, as a fraction of volume, which tells you whether the allowance is calibrated.
- • Window state size and open window count, which is where a watermark stall shows up as a resource problem.
- • Any event-time windowed computation — without a watermark, no window can ever close.
- • Bounded replays of historical data, where a perfect watermark makes recomputation exactly correct.
- • Pipelines with several inputs of differing speed, where the minimum rule gives a principled definition of joint completeness rather than an arbitrary one.
- • Sources with wildly unpredictable delay, where no allowance is simultaneously fast enough and complete enough and the strategy becomes a source of incidents.
- • Jobs with frequently idle inputs, where you are choosing continuously between a stall and manufactured lateness.
- • Low-latency requirements, where the allowance is pure added delay on every result.
- • Processing-time windows, which need no watermark at all and answer a different question — legitimate for pipeline monitoring.
- • Source-declared completeness markers, converting the heuristic into a fact wherever the producer can cooperate. Ask for this before building an estimator.
- • Trigger-based early firing with later correction, which decouples "emit something now" from "know it is complete" and removes most of the pressure on the watermark.
- • Batch recomputation over stored records, where completeness is known because the input is bounded, at the cost of latency.
A guess about time, made precise enough to act on
What people believe, and what is true
The watermark tells you the window is complete.
It tells you the processor is willing to *assume* completeness. Records can and do arrive afterwards; that is what Late Events: The Window Already Fired handling is for.
Watermarks are computed from wall-clock time.
They live on the event-time axis and are derived from the timestamps in the data. Some strategies use wall clock to advance during idleness, which is a fallback, not the definition.
If windows are not firing, the job is broken.
Most often the job is healthy and one input has stalled the minimum. Lag is fine, CPU is low, nothing errors, and nothing can fire.
A bigger allowance is safer.
It delays every result by that amount and grows window state proportionally. It buys completeness with latency and memory, both of which have their own failure modes.
A watermark going backwards would fix late data.
It would un-finalise windows that downstream has already acted on. Monotonicity is what makes finality mean anything, which is why lateness is handled by policy instead.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A watermark is the pipeline saying "I think I have seen everything up to this time", so windows before it can close. It is a guess, and records can still show up afterwards.
Practical
Pick an out-of-orderness allowance from the measured skew distribution. Turn on idleness detection and accept the lateness it creates. Alert on watermark age and expose per-input watermarks — that single view diagnoses the most common stall. Validate event timestamps at ingest so one future-dated record cannot advance the watermark past everything.
Advanced
A watermark is a failure detector for data rather than for nodes. Both answer an unfalsifiable question — "is anything more coming?" versus "is that node still alive?" — with a timeout, and both are therefore heuristics that trade a false-positive rate against a delay. The minimum-across-inputs rule is the same construction as waiting for a quorum, and idleness detection is the same concession as proceeding without a slow member. Recognising the shared structure predicts the failure modes without having to learn them separately: too aggressive means acting on incomplete information, too conservative means never acting, and there is no setting that is right under all conditions because the underlying question is not decidable.
Apply it
- 🔧 Stop a producer on one partition and observe the job stop firing windows while every health metric stays green. Then enable idleness detection and observe the lateness it creates.
- 🔧 Implement the minimum-across-inputs watermark with monotonic clamping and idleness, and write the test that catches a non-monotonic advance.
- ⚡ Window state has grown to 8 GB and no windows have fired in four hours. Diagnose from the per-input watermark table alone.
- ⚡ A quiet overnight partition means the first records each morning are dropped as late. Design a fix that does not stall the job at night.
- 💬 What is a watermark, and what does it guarantee? Be careful with the second half.
- 💬 Your job stopped emitting windows. Lag is zero, CPU is low, no errors. What do you check first?
- 💬 One device has its clock set to 2106. Walk me through what happens to your pipeline.