Late Events
An event happened at 10:00 and arrived at 10:07. The 10:00–10:05 window was already emitted. What happens next is a policy decision, and most platforms have made it by accident.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
An event arrives after the window it belongs to has already been emitted — is it counted, dropped, or does the previous answer change?
Anyone reading a number for a period that is recent enough to still be receiving data. They need to know one thing you almost never tell them: whether the number they are looking at is still moving, and if so, by how much it might still move (The Freshness SLO).
One event whose arrival_time is greater than the end of the window its event_time places it in. Lateness is a property of the pair, not of the event: the same record is late for a five-minute window and perfectly punctual for a daily one.
Close the window when its end time passes and emit. Anything that arrives afterwards missed the boat — and since almost nothing arrives late in testing, this behaves perfectly until it does not.
A mobile client that was offline uploads three hours of events at once. Every one of them belongs to a window that has already emitted, and every one is silently discarded. The job reports no errors and the daily total is quietly low (Missing Rows).
- A mobile client that was offline uploads three hours of events at once. Every one of them belongs to a window that has already emitted, and every one is silently discarded. The job reports no errors and the daily total is quietly low (Missing Rows).
- A connector restarts and catches up. The catch-up records are all late relative to windows that closed while it was down, so an outage that lasted twenty minutes causes a permanent hole rather than a delay (Ingestion Failure & Recovery).
- Someone widens the lateness allowance to stop the loss. State grows by the same factor, checkpoints slow down, and the memory problem arrives a month later with no obvious connection to the change (Streaming State).
- The stream and the nightly batch job disagree about yesterday, because batch saw the late records and the stream did not. Two numbers, both defensible, and no way to say which is correct without knowing the lateness policy (Two Dashboards, Two Numbers).
- A late record is accepted and the window re-emits, but the sink appends rather than upserts, so the period now has two rows and the dashboard sums both (Duplicate Rows).
What is actually happening
- A record is late when the watermark has already passed the end of the window its event time assigns it to. Lateness is therefore defined by the watermark, not by the wall clock — a slow watermark makes nothing late, and a fast one makes everything late (Watermarks).
- The allowed lateness is how long a window is kept open after the watermark passes its end. During that period a late arrival still updates the window and the result is re-emitted; after it, the window state is purged and there is nothing left to update.
- Assignment is always by event time, never by arrival. An event that happened at 10:00 and arrives at 10:07 belongs to the 10:00–10:05 window and nowhere else. Whether it is *counted* there depends entirely on whether that window is still open (Event Time).
- Once a window has been purged, a late record has three possible destinations and no fourth: dropped, routed to a side output for separate handling, or passed downstream as a correction that the sink applies to an already-published result.
- Re-emitting a window means the sink receives the same key twice with different values. That is correct behaviour and it is only safe if the sink upserts. An appending sink turns every correction into a duplicate (Upserts and Merges).
- The trade is exact and unavoidable: every minute of allowed lateness is a minute of extra state held per open window, and a minute of extra delay before the result is final. There is no configuration that improves both.
- Distributed Systems owns the mechanism underneath this — how a watermark is derived, why it can only ever be an estimate, and what a window firing means when the processor is itself a set of machines that disagree about the time (Watermarks: A Guess About Time, Made Precise Enough to Act On). This lesson owns the consequence for a reported number: whether the figure a consumer is looking at is final, how much it can still move, and whose job it is to say so.
The event that happened at 10:00 and arrived at 10:07
runStream(LATE_EVENT_SCENARIO, ...) in src/de/sim/stream.ts, whose behaviour is asserted in scripts/de-sim.test.ts for both the zero-allowance and wide-allowance cases. Clock labels are teaching positions, not measured latencies.Six events, five minutes to a window, one of them delayed. Five arrive in order and land where you would expect. The sixth happened at 10:00 — earlier than any of the others — and did not reach the processor until 10:07, by which time the 10:00–10:05 window had ended two minutes earlier.
The first thing to settle is where it belongs, and there is no ambiguity: the 10:00–10:05 window, because assignment is by event time and its event time is 10:00. Assigning it to 10:05–10:10 because that is when it arrived is the specific error event-time processing exists to prevent, and it would move a record into a period in which nothing happened.
The second thing to settle is whether it is *counted* there, and that is a policy question with two answers. When the late record arrived, the highest event time seen was 10:06, so with no lateness allowance the watermark stood at 10:06 — already past the window's end — and the window had been purged. With an allowance wide enough to keep the watermark behind 10:05, the window was still open and the record updates it, causing the result for 10:00–10:05 to be emitted again with a larger value.
That is the whole of §76, and it is worth stating what it costs. The version that counts the record holds every window open longer, so it uses more memory and produces its final answer later. The version that drops it is cheap, fast, and quietly wrong by an amount nobody measures unless a counter exists.
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| a | 10:01 | 10:01 | 10:00–10:05 On time. Watermark advances to 10:01. |
| b | 10:02 | 10:02 | 10:00–10:05 On time. Watermark advances to 10:02. |
| c | 10:04 | 10:04 | 10:00–10:05 On time. Watermark advances to 10:04 — still inside W1, so W1 has not closed. |
| d | 10:06 | 10:06 | 10:05–10:10 The watermark reaches 10:06 and passes W1's end. With no allowance, W1 emits here and is purged. |
| late | 10:00 | 10:07 | 10:00–10:05 The §76 event. It belongs to W1 by event time. Counted there if the allowance kept W1 open; dropped if not. It carries an old event time, so it does not advance the watermark at all. |
| e | 10:08 | 10:08 | 10:05–10:10 Watermark advances to 10:08. W2 is still open; W1 is long gone either way. |
Watermark position at the moment the late event arrived, with no lateness allowance: the highest event time seen so far, which is 10:06 from event d. W1 ends at 10:05, so 10:06 is already past it and the record is dropped. Widen the allowance and the watermark sits further back, W1 is still open, and the same record is counted where it belongs.
Four things you can do with it, and what each costs
There are exactly four responses to a record that arrives after its window closed, and every platform has chosen one — usually the first, usually without a decision being taken, because dropping is what happens when nothing is configured.
The choice is not a preference. It follows from two facts you can look up: how late your data actually gets, and whether the number is used for something that can tolerate being revised. A finance metric that is signed off cannot be revised after the fact, so its completeness must be bought before publication; an operational chart can be revised freely and should take the cheap option.
Note that the last two options are complementary rather than alternative. The strongest practical design is a modest allowance for the common case, a side output for the tail, and a periodic batch pass that merges the side output into history — three mechanisms, each cheap, covering a range no single allowance could.
How complete does this period have to be, and can the published answer change after it is first published?
when The metric is operational, the loss is small and measured, and revision is not worth the machinery.
cost Silent, permanent, unquantifiable data loss unless a counter exists. Acceptable only with the counter, which converts "we lose some" into "we lose this much" (Pipeline Metrics).
when The lateness distribution has a tail you can afford to hold in state, and the sink can upsert.
cost State multiplied by allowance ÷ window size, a later final answer, and re-emissions the sink must absorb (Streaming State).
when You want the loss to be recoverable without holding windows open, which is almost always.
cost A second dataset with its own owner, retention and monitoring — and it is worthless if nobody ever merges it (Late-Arriving Data).
when Consumers can handle a value that changes, and the sink applies updates by key.
cost Every consumer must be idempotent and must tolerate revision. A downstream system that snapshots the value on read will hold a stale copy forever (Upserts and Merges).
when The tail is long — hours or days — and holding it in streaming state is not viable.
cost A second implementation of the metric, which must agree with the streaming one by construction rather than by hope (Batch and Streaming Unification).
Making the loss visible
Everything about late events is manageable except the silence. A dropped record produces no error, no failed assertion, no lag and no gap in a chart — the number is simply a little smaller than the truth, forever, by an amount that varies with upstream conditions.
That makes instrumentation the first intervention rather than the last. A counter of dropped-as-late records costs a line of configuration and converts the entire failure class from invisible to obvious. If you take one thing from this lesson into a running system, take that counter.
The checks below are ordered by how much they buy relative to what they cost. Notice that the last two require infrastructure that only exists if somebody built it — a batch recomputation, a source-side reconciliation — while the first two are nearly free. This is the usual shape of data quality work: the cheap checks find the loud problems, and finding the quiet ones is a project.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Counter: records dropped as too late, per source | We discarded exactly this many records because their window had closed. | Every drop caused by the lateness policy, immediately, with attribution to a source. | Records that never arrived at all, and records that were late but still counted — it measures the policy, not the completeness (Missing Rows). |
Histogram of arrival_time − window_end for late arrivals | How late our late data actually is. | An allowance that is badly sized in either direction, and a producer whose buffering behaviour has changed. | Anything about records outside the window that were dropped before being measured, if the instrumentation sits after the drop rather than before it. |
| Streaming result versus batch recomputation for a closed period | The stream saw everything the raw data contains for this period. | The aggregate effect of dropped late records, state lost across a restart, and window boundaries that do not mean what was assumed. | A definition error shared by both implementations, and anything about periods still open (Reconciliation). |
| Reconciliation against the source system | Everything that happened reached us. | Data that never arrived — a producer that stopped, a connector gap, an upstream filter. | Everything about correctness of values; a period can reconcile perfectly on counts and be wrong in every amount (The Dimensions of Data Quality). |
The first row is nearly free and finds the failure this lesson is about. The rest exist because a dropped-record counter proves the policy is working and says nothing about whether the data ever showed up.
How to build it
Most important first.
- Measure your lateness distribution before choosing anything.
ingestion_time − event_timeper source, as a distribution over weeks, is the input to this decision, and choosing an allowance without it is guessing (Percentiles: Which One, and How Many Users Is That?). - Set the allowance from the distribution's tail that you can afford to hold, not from its maximum. Covering the extreme tail means holding every window open that long, which is usually the wrong trade (Streaming State).
- Route what falls outside the allowance to a side output rather than dropping it. A late-record stream is cheap to write, makes the loss measurable, and gives a batch correction something to work from (Late-Arriving Data).
- Make the sink idempotent on the window key, so a re-emission overwrites rather than adds. This is the precondition for accepting late data at all (Idempotent Data Pipelines).
- Publish two numbers rather than one where it matters: the provisional current value and the point at which the period becomes final. A consumer who knows a number is still moving behaves completely differently from one who does not (Dataset Documentation).
- Handle the long tail in batch. The right architecture for genuinely late data is usually a stream with a modest allowance plus a periodic batch recomputation over raw storage that corrects the record — not an enormous allowance that makes the streaming job unoperable (Reprocessing vs Retrying).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Within the allowed lateness: the window will include the record and the updated result will be emitted. Nothing promises that a consumer read the earlier version, or that it will read the update.
- Outside the allowed lateness: the record is not in the window. If a side output was configured, it exists somewhere; if not, it is gone, and nothing in the pipeline records that it existed (Missing Rows).
- Emitting a corrected window guarantees only that the correct value was *sent*. Whether the published number changes depends on the sink's semantics — upsert corrects, append double-counts (Atomic Publish).
- What is explicitly not guaranteed: that the number a consumer saw at any moment was complete; that all late data is captured by any finite allowance; or that two consumers reading at different times saw the same value for the same period.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Count records dropped as too late, per source, and alert on any non-zero value. This is the single most important counter in the module: silent dropping is its characteristic data loss, and nothing else in the pipeline reports it.
- Reconcile a closed period against a batch recomputation from raw storage. The difference between the two is exactly the data the stream discarded, which turns an invisible loss into a measured one (Reconciliation).
- Both miss the case where the late records were never produced or never arrived at all. Neither a dropped-record counter nor a reconciliation against your own raw storage can see data that did not reach you — only reconciliation against the *source* can (Missing Rows).
- Allowed lateness converts an unbounded, invisible incompleteness into a bounded, stated delay. That is the real product here: not more correct data, but a *knowable* time at which the number stops moving.
- A window's result is provisional from its end until the allowance expires, and final afterwards. Consumers should be told which of the two states they are looking at, because the correct reaction differs completely (Freshness Monitoring).
- The relationship is linear and unforgiving: doubling the allowance doubles the time to a final answer and roughly doubles the number of windows held open. Freshness and completeness trade directly against each other and against memory (Cost vs Freshness).
- Changing the allowed lateness changes the metric's definition. Periods computed under a five-minute allowance are not comparable with periods computed under an hour, and nothing in the schema records which one produced which range (Semantic Changes).
- Record the policy as metadata on the dataset, versioned, with the date of each change. This is one of the clearest cases in the domain where the meaning of a number lives outside its schema (Dataset Documentation).
- Adding a side output for late records is a compatible change for existing consumers and creates a new dataset with its own contract, owner and retention — it is a data product, not a debug log (Data Products).
- Late records captured in a side output can be merged into the affected periods by a batch job keyed on window and group. This is the standard correction path and it is why the side output is worth the trouble (Late-Arriving Data).
- Where no side output exists, recovery means replaying the log for the affected range through a job with a wider allowance, which works only within retention and only if the sink upserts (Replay from the Log).
- Corrections must be published atomically per period. Applying a correction row by row makes a consumer able to observe a half-corrected period, which is worse than either the wrong number or the right one (Atomic Publish).
What can go wrong
- Silent dropping with no counter, so the loss is discovered by a finance reconciliation months later and cannot be quantified retroactively.
- An allowance so wide that state growth makes checkpoints slow and restores long — the mitigation for lateness becoming the cause of the outage (Streaming State).
- A corrected re-emission appended rather than upserted, turning every late record into a double count (Duplicate Rows).
- A future-dated record advancing the watermark past everything real, so a large share of genuinely punctual data is classified as late and dropped at once (Watermarks).
- The mitigation failing: a side output written to a location with no owner, no monitoring and no retention, so the recovery data expires before anyone reads it.
- "Late events are rare." They are rare in the tests and normal in production, because production has mobile clients, retries, partner feeds and connector restarts. The correct assumption is that lateness exists and has a distribution you have not measured.
- "A bigger allowance is safer." It is more complete and less operable, and past a point it makes the job fail in a way that loses everything rather than a little. The safe design is a modest allowance plus a batch correction (Reprocessing vs Retrying).
- "Late means the record is wrong." The record is correct and the timing is inconvenient. Discarding it is a decision to be less accurate in exchange for being more timely, and it should be made deliberately rather than by default.
- "The batch job and the stream disagree, so one has a bug." They may both be correct under different lateness policies. Two implementations of a metric with different completeness rules are two different metrics (Two Dashboards, Two Numbers).
Operating it
- Dropped-as-late count, per source and per window definition. Non-zero is an incident, not a statistic.
- The lateness distribution as a histogram:
arrival_time − window_endfor records that arrive after their window ended. This says directly whether the current allowance is sensible (Histograms: A Distribution You Can Afford to Keep Forever). - Count of window re-emissions, which measures how much correction is actually flowing and whether the sink is being asked to upsert at a rate it can sustain.
- The gap between the streaming result and the batch recomputation for the most recent closed period, tracked as a series rather than checked ad hoc (Reconciliation).
- At 10x volume, lateness behaviour is unchanged: it is a property of transport delay, not of throughput.
- At 10x source count the distribution becomes multi-modal — seconds for web, minutes to hours for mobile, a day for a partner feed — and a single allowance is wrong for most of them. Per-source windowing, or per-source jobs, becomes the answer.
- At 100x, the tail thickens because there are more independent producers, more retries and more transient failures. An allowance chosen from a small deployment's tail will drop a growing share as the fleet grows (Percentiles: Which One, and How Many Users Is That?).
- Allowed lateness costs state directly: the number of windows held open per key is the allowance divided by the window size, so it is a straight multiplier on the largest cost term in a streaming job (Streaming State).
- It also costs write amplification at the sink, because every correction rewrites a row that was already written. For a warehouse this is a merge; for a lake table format it is a rewrite of the affected files (Open Table Formats).
- The alternative — a modest allowance plus a batch correction pass — moves cost from continuously-held memory to periodically-scanned storage, which is usually the cheaper shape and always the more operable one (Cost vs Freshness).
- A wide allowance buys completeness and costs memory, checkpoint duration, restore time and time-to-final-answer, all in direct proportion. A narrow one buys operability and costs silently discarded data.
- A side output buys measurable, recoverable loss and costs another dataset to own, monitor, retain and eventually merge — which is real work that is usually deferred until it is too late to matter.
- Publishing provisional results buys freshness and costs consumer trust when numbers move. The cost is worth paying only if the provisional state is labelled; unlabelled moving numbers destroy confidence faster than stale ones (Trusting Data).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALAny system that computes over a time period must decide what to do with data that arrives after the period was computed. Batch pipelines face the identical question and answer it by recomputing partitions rather than by holding windows open (Late-Arriving Data).
- SIMULATEDThe timeline in this lesson comes from
src/de/sim/stream.tsand is pinned byscripts/de-sim.test.ts; the times are positions on a teaching timeline rather than measurements, and the model collapses watermark delay and allowed lateness into one knob where real engines expose two. - ENGINE-SPECIFICFlink separates the watermark strategy from a per-window
allowedLatenessand can route late records to a side output; Spark Structured Streaming exposes a single watermark that governs both state retention and lateness, and drops late records with no side-output equivalent. The same configuration intent therefore requires different mechanics and offers different recovery options.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why arrival order and occurrence order diverge in the first place, and why no amount of engineering removes the possibility of an arbitrarily delayed message.