The question this answers
My hourly counts are wrong after an outage. Which clock was I counting by?
Processing-time windows are complete the instant they close and are trivially deterministic, but their contents depend on system behaviour and are not reproducible across a replay. Event-time windows describe what actually happened and are reproducible across replays, but can never be known to be complete — only estimated, which is what Watermarks: A Guess About Time, Made Precise Enough to Act On exist to do.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The processor knows its own wall clock and the timestamps carried inside the records it has received. It does not know whether any more records exist for a past event-time window — there is no message announcing that a source is finished, and a phone that was offline may still upload yesterday. Completeness for event time is unknowable, and every design here is a way of managing that rather than solving it.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The two clocks, and the gap between them
Event time is when the thing happened in the world: the tap on the phone, the sensor reading, the payment authorisation. It is a property of the event and it never changes, no matter how many times you reprocess. Processing time is when a record arrived at the operator computing over it. It changes on every replay, every restart, every rebalance and every deploy.
The gap between them is never zero and it is never constant. Network transit, broker buffering, consumer lag, a batch producer that uploads every five minutes, a mobile client that was in a tunnel — each contributes. A gap of milliseconds is normal; a gap of hours during a backlog drain is normal; a gap of days for a mobile client is normal. There is no upper bound you can rely on, which is the fact everything else follows from.
The consequence is a fork in the road that every stream design walks past, usually without noticing. Group records by processing time and you are measuring your system’s behaviour. Group by event time and you are measuring the world. Both are legitimate. Only one of them is what a business dashboard is asking for, and it is essentially always event time.
What goes wrong, specifically
The failures here are not vague inaccuracy; each has a precise shape, and recognising the shape is how you diagnose it from a graph alone.
Backlog drain distortion. The consumer is down for two hours. On restart it processes two hours of records in five minutes. Processing-time hourly counts show two near-empty hours followed by one enormous spike. Nothing happened in the world to cause that; you plotted your outage. Event-time windows show flat, correct hours — the outage is invisible in the data, which is exactly right.
Replay non-determinism. Reprocess last month with processing-time windows and every record lands in today. The output is not merely different from the original, it is meaningless. Event-time windows produce the same answer on every replay, which makes replay a usable recovery tool rather than a destructive one.
Boundary leakage. A user session spanning 09:58 to 10:03 is split across two processing-time windows in a way that depends on how fast the consumer happened to be. Session and funnel analysis computed this way is not reproducible, and two runs disagree with each other.
Timezone and clock trust. Event time usually comes from a client, and clients lie: wrong timezones, unset clocks, deliberately manipulated device time. An event-time pipeline must decide whose clock to trust — client timestamp, server ingest timestamp, or both recorded separately — and the honest answer for anything billable is to trust the server and keep the client’s claim as data rather than as truth. See Two Timestamps Are Not an Ordering and Clock Skew: The Gap You Cannot Measure From Inside for why the client’s number is not a fact.
| Processing time | Event time | |
|---|---|---|
| What it measuresprotocol | Your system’s throughput and lag | What happened in the world |
| Window completenessprotocol | Known immediately when the window closes | Never known; only estimated |
| Deterministic across replayprotocol | No — results change every run | Yes, given the same records |
| Effect of a 2-hour outageprotocol | Two empty hours then a spike | No effect on the shape at all |
| Requires buffering stateprotocol | No | Yes — windows stay open for allowed lateness |
| Trustsassumption | Your clock only | Whichever clock stamped the event |
| Right fortypical | Alerting, SLOs, monitoring the pipeline | Billing, analytics, reporting, anything a person compares over time |
Event-time windows need state, and state has a cost
A processing-time window is cheap: records arrive, you add them to the current bucket, the clock ticks over, you emit and forget. Memory is bounded by one window.
An event-time window cannot close when its end time passes, because records belonging to it may still arrive. So the processor must hold open every window that might still receive data — for the configured allowed lateness — which means memory proportional to key cardinality × open windows. Allow a day of lateness on hourly windows keyed by user and you are holding 24 open windows per active user. This is the dominant cost of event-time processing, and it is the reason allowed lateness is a resource decision as much as a correctness one.
It also means the processor is now stateful in a way that matters for Rebalancing: Everyone Stops So the Partitions Can Move: partition reassignment moves that window state, or forces it to be rebuilt. Frameworks handle this with checkpointed state backed by a changelog, which works and is another thing to operate. A stateless consumer that reassigns instantly is a genuine advantage of processing-time windows, and occasionally decisive for a monitoring pipeline.
1// PROCESSING TIME: stateless, immediate, wrong after any lag2onRecord(r):3 bucket = floor(now() / HOUR) // OUR clock4 counts[bucket] += 15// Replay this tomorrow and every record lands in tomorrow's bucket.6 7// EVENT TIME: stateful, delayed, reproducible8onRecord(r):9 bucket = floor(r.eventTime / HOUR) // the RECORD's clock10 if bucket < currentWatermark - allowedLateness:11 sideOutput(r) // too late; see [[late-events]]12 else:13 counts[bucket] += 114 openWindows.add(bucket)15 16onWatermarkAdvance(w):17 for bucket in openWindows where bucket + HOUR + allowedLateness <= w:18 emit(bucket, counts[bucket]) // fire, and only now free the state19 openWindows.remove(bucket)20 21// Memory held: keys x open windows. Allowed lateness is a memory decision.Choosing, and recording both
The rule that survives contact with production: use event time for anything describing the world, and processing time for anything describing the pipeline. Revenue by hour, active users per day, conversion funnels — event time. Records processed per second, consumer lag, alert on ingest stalling — processing time, because there the system *is* the subject.
And regardless of which you window by, record both timestamps on every record. The difference between them is one of the most useful metrics in a streaming system: it is the true end-to-end lag, it tells you how much lateness to allow, and it makes an unexplained gap attributable to a specific stage. A pipeline that keeps only one timestamp cannot answer "was this slow, or did it happen a while ago?" — a question that comes up in every incident.
One more discipline worth adopting: make the event-time field explicit and validated at the boundary. A record with no event time, a null event time, or an event time in the year 2106 will silently ruin a windowed aggregate, and by the time the dashboard looks wrong the offending record is far behind the current offset. Reject or quarantine at ingest, where the producer is still identifiable.
Key points
- Event time is when it happened and never changes; processing time is when you saw it and changes on every replay.
- The gap between them is unbounded, and a backlog drain plots your outage rather than the world.
- Only event-time windows are reproducible across a replay, which is what makes replay a usable recovery tool.
- Event-time windows must stay open for allowed lateness, so memory is key cardinality times open windows — a resource decision.
- Record both timestamps on every record; their difference is the true end-to-end lag and the input to every lateness decision.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The producer stamps the record with the time the event occurred, ideally alongside a server-assigned ingest timestamp.
- • The record travels through the broker and is fetched by the processor at some later, unbounded processing time.
- • The processor assigns the record to a window using the chosen clock: the record’s event time, or its own wall clock.
- • For event time, the window is held open while the watermark advances toward its end plus allowed lateness.
- • When the watermark passes that threshold, the window fires and its state is released.
- • A record arrives with no event-time field, or a null one, and is silently assigned to the wrong window.
- • A client clock is wrong, placing events in windows far in the past or future.
- • A backlog drain compresses hours of records into minutes, distorting every processing-time aggregate.
- • Allowed lateness is set generously and window state exhausts memory.
- • A replay produces different results from the original run, making the recomputation useless for correction.
- • Post-outage spike: the operator sees two flat hours and one hour at 300% of normal on the revenue dashboard. Finance asks what happened; nothing happened, the consumer was down and the graph is plotting the recovery.
- • Replay produces garbage: a month is reprocessed to fix a bug and every record lands in today’s window. The corrected output is unusable and the original is already overwritten.
- • Future-dated records: a device with a clock set to 2106 creates a window that will never close, holding state forever. Memory grows linearly and the leak is attributed to the framework.
- • Silent null-timestamp bucketing: records with a missing event-time field default to epoch zero or to the current time depending on the framework. Counts are subtly wrong with no error, and the offending producer is only identifiable at ingest.
- • State explosion from allowed lateness: a config change from one hour to one day of lateness multiplies open windows by 24 and the job starts OOM-looping during the next traffic peak.
- • None between partitions for the assignment itself — each record carries its own time, so bucketing is a local decision.
- • Coordination reappears in deciding *completeness*: an event-time window spanning several partitions cannot fire until every partition has advanced past it, which is why a watermark is a minimum across sources and why one idle partition stalls everything.
- • Processing time requires no such agreement, which is precisely why it is complete immediately and precisely why it is meaningless across a replay.
- • Event-time results are unaffected in shape by consumer downtime; the records land in their correct windows whenever they arrive, provided those windows are still open.
- • Processing-time results are permanently distorted by any downtime and cannot be corrected by reprocessing.
- • Window state held for open event-time windows must survive restarts, or every restart silently discards partial aggregates.
- • Detect: monitor the distribution of event-time-to-processing-time skew. A shifting distribution is the earliest signal that windows will start firing without their data.
- • Contain: after an outage, do not trust processing-time-based aggregates for the affected period; mark them rather than publishing them.
- • Recover: recompute event-time aggregates by replaying the affected range, which is possible precisely because event time is stable.
- • Reconcile: compare recomputed event-time aggregates against what was originally emitted, and republish corrections through the same path.
- • Verify: replay the same range twice and confirm identical output. If it differs, something in the pipeline is still using processing time.
- • Event-time skew distribution (processing time minus event time) at p50, p95, p99 and max — the single most valuable metric in a streaming pipeline.
- • Count of records with missing, null, or implausible event times, attributed to producer.
- • Open window count and window state size per key, which predicts memory pressure before it becomes an OOM.
- • Windows fired per interval against windows expected, which reveals a stalled watermark.
- • Records arriving after their window fired, which is the direct measure of whether allowed lateness is calibrated.
- • Any aggregate a human will compare across time periods: revenue, active users, conversion, latency percentiles of business events.
- • Any pipeline where replay is part of the recovery story, which is most log-based pipelines.
- • Any source with genuinely delayed delivery: mobile clients, IoT, batch uploads, third-party webhooks.
- • Real-time alerting, where you want to know about the last 60 seconds of *system* behaviour and waiting for completeness defeats the purpose.
- • Very high key cardinality with long allowed lateness, where window state becomes the dominant cost of the whole pipeline.
- • Sources with untrustworthy clocks and no server-side ingest timestamp, where event time is fiction dressed as data.
- • Ingestion time — the timestamp assigned when the record entered the system. A middle ground: stable across replay, immune to client clock lies, but still wrong about when the event actually occurred.
- • Processing time with an explicit statement that the metric describes the pipeline, which is honest and correct for monitoring.
- • Batch recomputation over stored records on a schedule, which gives exact event-time answers with high latency and none of the streaming state cost.
- • Dual output: emit a fast processing-time approximation for immediacy and a corrected event-time result later, which is the lambda-style compromise and costs two implementations.
Two clocks: when it happened, and when you saw it
What people believe, and what is true
Events arrive in the order they happened.
They arrive in whatever order the network, partitions, retries and client connectivity produced. Out-of-order arrival is the normal case, not the exception.
The difference is small enough to ignore.
It is milliseconds in the steady state and hours during a backlog drain — and the drain is exactly when someone looks at the dashboard.
Processing time is fine because our lag is low.
Your lag is low until it is not. The design is correct only under a condition you do not control, and it fails precisely during incidents.
Event time is just a field on the record.
Choosing it commits you to open windows, buffered state, a lateness policy and a watermark strategy. It is an architecture, not a column.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Every record has a "when it happened" and a "when we saw it". Group by the first to describe the world, by the second to describe your pipeline. Confusing them makes an outage look like a business event.
Practical
Stamp both timestamps at ingest and validate the event-time field there. Window business metrics by event time and pipeline metrics by processing time. Monitor the skew distribution; it tells you what allowed lateness to set. Budget memory for key cardinality times open windows before choosing that number.
Advanced
This is There Is No Global Clock arriving in the analytics layer. There is no global observer who knows when an event occurred, so event time is a claim made by a participant, and there is no global observer who knows when a stream is complete, so window completeness is unknowable rather than merely unknown. Every practical system therefore substitutes an estimate — a watermark — and accepts a bounded probability of being wrong, exactly as a failure detector does for liveness. The design question is not how to get certainty; it is how much lateness to tolerate before acting, and what to do about the records that prove you acted too early.
Apply it
- 🔧 Compute the same hourly aggregate both ways over a replayed dataset and diff the outputs. Then stop the consumer for an hour and diff again.
- 🔧 Plot the event-time skew distribution for a real topic and use its p99 to justify an allowed-lateness setting.
- ⚡ A mobile client uploads a day of events after a flight. Trace what each windowing choice does with them.
- ⚡ A device with a clock set to 2106 arrives in your stream. Describe what happens to memory and how you would prevent it at ingest.
- 💬 Your hourly revenue chart shows two empty hours and then a huge spike. Nothing happened in the world. Explain.
- 💬 Why can you never know an event-time window is complete?
- 💬 Which timestamp would you use for a billing pipeline, and whose clock do you trust?