Ingestion Time
The moment a record entered the processing system, stamped by one clock the platform controls — the timestamp that makes lag measurable and replay stable without pretending to know when anything happened.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
When did this record become my platform's responsibility, and what can I hold myself accountable for from that moment on?
The team that owns the pipeline and the SLO attached to it. Ingestion time is the boundary between "the producer was slow" and "we were slow", and without it every freshness conversation is an argument in which nobody can prove anything (The Freshness SLO).
One record, at the moment the platform accepted it — typically stamped by the broker when the record was appended to a partition. One clock, one stamping point, one value that never changes afterwards.
Two timestamps are enough: when it happened and when we processed it. Adding a third is bookkeeping for its own sake, and the difference between arriving and being processed is small enough to ignore.
End-to-end freshness is missing its SLO and nobody can say whether the producer was late or the consumer was slow, because the only two timestamps available span both (Pipeline SLOs).
- End-to-end freshness is missing its SLO and nobody can say whether the producer was late or the consumer was slow, because the only two timestamps available span both (Pipeline SLOs).
- A replay reprocesses records from the log. Processing time changes on every replay, so any lag figure derived from it is meaningless afterwards — while ingestion time, stored with the record, still says exactly when it arrived.
- A mobile fleet with unreliable clocks produces event times that cannot be trusted for ordering at all. There is no platform-controlled timestamp to fall back on, so the choice is between a bad clock and no clock (Event Time).
- A retention policy needs to expire data by when it was received, and the only available times are one the producer chose and one that does not survive a restart (Data Retention).
- A partner feed carries no per-record timestamp whatsoever. Without an ingestion stamp there is no time dimension on that dataset at all, and nothing can be windowed or expired.
What is actually happening
- Ingestion time is assigned by the platform, at the boundary where a record is accepted and made durable — normally the broker appending to a partition, sometimes the collector in front of it (Message Brokers: Log-Shaped and Queue-Shaped).
- Because it is stamped once and stored with the record, it is stable across replays. That is the property that separates it from processing time: reprocessing does not change it, so anything computed from it is reproducible (Processing Time).
- Because it is assigned by a clock the platform controls, it is trustworthy in a way event time is not, and effectively monotonic within a partition — records are appended in order and stamped in that order (Topics and Partitions).
- Because it describes arrival rather than occurrence, it is wrong for business questions. A batch of buffered events uploaded on reconnect all share roughly one ingestion time and belong to a spread of event times (Late Events).
- The arithmetic that makes it valuable:
ingestion_time − event_timeis producer-and-transport lateness, andprocessing_time − ingestion_timeis your pipeline's own delay. Neither is computable without all three timestamps present, and together they decompose end-to-end freshness into terms with owners (Latency Budgets: Spending 200 Milliseconds on Purpose). - A broker may be configurable to overwrite the producer's timestamp with its own append time. That is a per-topic decision with a real consequence: it makes ordering trustworthy and destroys event time unless the producer also carried it in the payload (Kafka as a Log, Not a Queue).
Three timestamps, three owners
A record accumulates timestamps as it travels, and each one is assigned by a different party with a different degree of trustworthiness. The producer says when it happened. The platform says when it arrived. The processor says when it handled it. Keeping all three is what turns "the dashboard is stale" into a question with a determinable answer.
The pipeline below shows where each is stamped and what each one is actually good for. The middle stage is the one usually missing, and its absence is why so many freshness discussions end with two teams each convinced the other is the problem.
Read the guarantees column carefully. Event time is the most useful and the least trustworthy; processing time is the most trustworthy and the least reproducible; ingestion time is the only one that is both assigned by a clock you control and stable across replays. That combination is narrow and it is exactly what an SLO needs.
- 1Action occurs
A user taps, a sensor reads, a row commits. The producer records
occurred_at.guarantees Describes the world. Reproducible on replay because it travels in the payload.
fails by A clock you do not control: drift, wrong zone, user-set, or stamped at serialisation rather than at the action (Event Time).
- 2Producer buffers and sends
Batches records, retries on failure, holds them while offline.
guarantees Nothing about timing. This stage is precisely the delay that
ingestion − eventmeasures.fails by Silent multi-hour buffering that looks identical to a pipeline outage from downstream (Late Events).
- 3Platform accepts
The broker appends the record to a partition and stamps
ingested_at.guarantees A clock the platform controls, stable across replays, monotonic within a partition, always present.
fails by Broker configuration overwriting the producer timestamp, which changes the meaning of a field with no schema change (Message Brokers: Log-Shaped and Queue-Shaped).
- 4Processor handles
An operator reads the record and stamps
processed_at.guarantees Always available; the correct clock for throughput, lag and timeouts.
fails by Changing on every replay, so nothing computed from it can be reproduced (Processing Time).
- 5Sink publishes
Writes the result where a consumer can read it.
guarantees Whatever the sink promises about atomicity — the last term in end-to-end freshness (Atomic Publish).
fails by A serving layer or BI cache adding a further delay that none of the three timestamps captures (Stale Dashboards).
Freshness a consumer experiences is the sum of all five stages. Only the middle three are yours, and only ingestion time marks where your part begins.
What each clock is and is not good for
The three time domains are not three approximations of one thing. They answer three different questions, and the reliable way to choose between them is to ask which question you are actually asking rather than which value is most convenient to obtain.
The matrix below is the compressed form of this whole group of lessons. The column worth staring at is "stable on replay", because it splits the three cleanly: event time and ingestion time survive reprocessing, processing time does not, and that single property decides whether history can ever be recomputed.
The second column worth staring at is "whose clock". Trust follows control: you control the platform clock, you can usually verify a server-side producer clock, and you cannot verify a device clock at all. For any number with commercial consequences, that ordering should decide which field the business logic depends on (Every Input Surface).
| Property | Event time | Ingestion time | Processing time |
|---|---|---|---|
| Answers the question | When did it happen? | When did it become ours? | When did we handle it? |
| Whose clock | The producer's — a device, an app server, a source database | The platform's — broker or collector | The processing machine's |
| Stable on replay | Yes — travels in the payload | Yes — stored with the record | No — changes on every run |
| Can be missing or malformed | Yes: absent, unparseable, in the future, wrong zone | No — assigned by the platform | No — always available |
| Ordering | Arbitrarily out of order; that is normal, not exceptional | Monotonic within a partition, unrelated across partitions | Monotonic per instance, skewed between instances |
| Right for | Business metrics, windows, backfills, period comparison | Freshness SLOs, retention, lag attribution, fallback ordering | Throughput, lag, timeouts, rate limits, emission triggers |
| Wrong for | Measuring your own pipeline's speed — it includes the producer | Any question about when something happened | Anything that will be reproduced, compared or audited |
| Characteristic failure | A wrong clock misplaces records silently and permanently | Mistaken for event time, so a producer stall reads as a business change | History cannot be recomputed, discovered months later |
The trap: ingestion time standing in for event time
Ingestion time is the most tempting substitute for event time, precisely because it is so much better behaved: never missing, never in the future, monotonic within a partition, assigned by a clock you trust. Every one of those properties is about your platform, and none of them makes it a description of the world.
The substitution is usually made without a decision being taken. A model is written against ingested_at because it is the reliable column; a dashboard is built on that model; and for months the two agree because the transport is fast. Then a producer buffers, or a connector stalls and catches up, and the chart shows a trough and a spike that describe transport rather than customers (Stale Dashboards).
The correct posture is narrow and easy to state: ingestion time is for statements about the platform — freshness, lag, retention, ordering fallback — and event time is for statements about the business. When a source genuinely has no event time, publish ingestion-time metrics with that limitation recorded in the dataset documentation, so the next person does not have to rediscover it during an incident.
Group by `date_trunc('hour', ingested_at)`. Never null, never out of order, always complete for a closed hour, and the chart matches event time on any healthy day.Group by `date_trunc('hour', occurred_at)`, hold the hour open for a stated lateness allowance, and keep `ingested_at` on the row so lateness can be measured and attributed.The two definitions differ by transport delay, which is near zero when nothing is wrong and large exactly when something is. The ingestion-time version therefore agrees with reality on the days nobody is looking and disagrees on the day someone is — and because it discards the placement information, the disagreement cannot be corrected afterwards.
Whether a broker stores the producer's timestamp or its own append time is a per-topic configuration in Kafka, and other services expose different fields with different names and different guarantees. Verify what your topics are actually configured to store before treating the transport timestamp as either an event time or an ingestion time; the safe design is to carry your own event time in the payload and not depend on the transport field at all.
How to build it
Most important first.
- Carry all three timestamps on every record, in every layer, all the way to the warehouse. They are a few bytes and they are the only way to attribute a freshness problem to a party (The Raw Landing Zone).
- Keep event time in the payload, where you control it, rather than relying on a transport-level timestamp field that a broker configuration can overwrite (Data Contracts).
- Define freshness SLOs on ingestion time, not on event time. You cannot commit to a number that includes a producer's buffering behaviour, and a producer-inclusive SLO is one you will breach for reasons outside your control (The Freshness SLO).
- Use ingestion time for retention and lifecycle decisions, because it is the moment your obligation began and it is the one timestamp an auditor can verify against your own logs (Data Retention).
- Use it as an ordering fallback where event time is untrustworthy, and say so in the dataset documentation — a dataset ordered by arrival is a different dataset from one ordered by occurrence (Dataset Documentation).
- Never window business metrics on it. It is closer to the truth than processing time and it is still a statement about your platform rather than about the world (Event Time).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Stability: assigned once, stored with the record, unchanged by any replay or reprocessing. This is what processing time lacks and what makes ingestion time usable in a reproducible computation.
- Availability: always present, because the platform assigns it. Unlike event time it cannot be missing, malformed, or in the future.
- Ordering within a partition: records are appended and stamped in the same order, so ingestion-time ordering agrees with offset ordering. Across partitions it agrees with nothing (CDC Ordering and Transaction Boundaries).
- What is explicitly not guaranteed: that it resembles when the event happened; that two brokers' clocks agree; or that it survives a change to broker timestamp configuration, which can silently switch the meaning of the field for new records only.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check for records where
ingestion_time < event_timebeyond a small tolerance. That means a producer clock ahead of the platform's, and it is the earliest and cheapest signal of the future-timestamp problem that wrecks watermarks (Watermarks). - Track the distribution of
ingestion_time − event_timeper source and alert on shape changes. A producer that started buffering differently shows up here and nowhere else (Distribution Tests). - Both miss the case where broker timestamp configuration changed, so ingestion time now means something different for records after a certain offset — with no schema change and no error. Only a discontinuity in the distribution reveals it (Semantic Changes).
- Ingestion time is what makes freshness measurable at all. Without it, "how fresh is this dataset" collapses into either "how old is the newest event time we hold" — which a producer stall makes look identical to a pipeline outage — or a processing-time figure that does not survive a replay.
- With it, the freshness question decomposes: producer-and-transport delay is
ingestion − event, platform delay isprocessing − ingestion, and serving delay is everything after. Each term has an owner and a separate remedy (Freshness Monitoring). - The number a consumer should be given is normally the ingestion-time freshness of the newest complete record, because that is the part the platform can actually commit to.
- The field itself rarely evolves; what evolves is its meaning. Moving the stamping point from a collector to the broker, or enabling broker-side timestamp overwrite, changes what the value represents for all subsequent records (Semantic Changes).
- That kind of change is invisible to every schema check because the name and the type are unchanged. It belongs in the data contract with a version, and the cutover offset or date should be recorded (Data Contracts).
- Adding ingestion time to a pipeline that lacked it is straightforward for new data and impossible for old: historical records cannot be retroactively stamped, so the field will have a start date that every query over history has to respect (Nullability & Defaults).
- Ingestion time is what makes a replay auditable. Because it is stored with the record, a reprocessed dataset can still report exactly when each record arrived, so a corrected run and the original are comparable on the axis that matters for SLO purposes (Replay from the Log).
- It also bounds what a replay can fix: records that never arrived have no ingestion time and are invisible to any check based on it. Missing data is found by reconciling against the source, never by inspecting what you have (Missing Rows).
- For backfills from object storage rather than from the log, preserve the original ingestion time rather than restamping. Restamping makes the backfilled range look like it arrived today and destroys the only record of when it really did (What Backfills Break).
What can go wrong
- A broker configured to overwrite producer timestamps, silently converting the event-time field into an ingestion-time field for a subset of topics.
- Clock differences between brokers, so ingestion-time ordering across partitions is subtly wrong in a way that offsets are not.
- Ingestion time used as event time by a downstream model, producing a metric that looks right until the day a producer buffers (Two Dashboards, Two Numbers).
- A collector that stamps on batch receipt rather than per record, so a thousand records share one ingestion time and the lateness distribution becomes uninformative.
- The mitigation failing: adding ingestion time and then defining the freshness SLO on it, while the consumer's actual complaint is about event-time completeness — the SLO is green and the consumer is still wrong (The Freshness SLO).
- "Ingestion time is close enough to event time." It is close while the transport is healthy and diverges exactly when it is not, which is when the number is being read. Two timestamps that agree on quiet days are not the same timestamp.
- "The broker timestamp is the event time." It is whatever the broker was configured to store, which is either the producer's claim or the broker's append time, and the configuration can change per topic without any schema change (Kafka as a Log, Not a Queue).
- "We can add it retroactively." New records only. Historical data cannot be stamped with a time it was not observed, so the field will always have a start boundary that queries must handle.
- "Three timestamps is over-engineering." Two timestamps cannot decompose a latency problem into terms with owners. The third is the difference between an incident with a cause and an incident with a meeting (Latency Budgets: Spending 200 Milliseconds on Purpose).
- Ingestion time is normally the correct clock for retention: your obligation to hold a record begins when you received it, and that moment is verifiable against your own broker logs in a way a producer-supplied timestamp is not (Data Retention).
- It is also the axis on which a deletion request is most reliably executed, because it identifies every copy that entered the platform within a range even when the event times are unreliable or absent (Deletion Requests).
Operating it
- Freshness as
now − max(ingestion_time)per dataset, which distinguishes a stopped pipeline from a quiet source in a way that an event-time-based freshness metric cannot (Freshness Monitoring). - The two decomposed lag terms, charted together: producer-and-transport lag and platform lag. When end-to-end freshness degrades, this pair says immediately whose problem it is (Latency Budgets: Spending 200 Milliseconds on Purpose).
- Count of records with
ingestion_timeearlier thanevent_time, which is a pure clock-skew detector and costs nothing to compute.
- At 10x volume, nothing changes. It is a per-record field with no aggregation behaviour.
- At 10x source count, it becomes more valuable rather than less: with many producers of varying reliability, ingestion time is the only common, trusted axis on which they can be compared.
- At 100x, partition-level clock differences that were irrelevant start to matter for anything that orders across partitions by timestamp — the correct response is to stop ordering across partitions by timestamp rather than to synchronise harder (Topics and Partitions).
- A timestamp per record: negligible storage, negligible compute, and it compresses extremely well because values within a file are near-identical and encode into a small delta (Dictionary, Run-Length, Delta and Bit Packing).
- The only meaningful cost is discipline — carrying it through every hop, including the ones where somebody will be tempted to drop it during a projection.
- The saving is large and indirect: an incident where the lag is attributable in one query instead of an afternoon of correlation across three systems (Debugging a Data Incident).
- Ingestion time buys attributable lag, stable replay-safe timing and a trustworthy fallback ordering, and costs a field per record plus the discipline to preserve it through every transformation.
- Using it as an ordering fallback buys a usable order where event time is untrustworthy, and costs correctness whenever the transport reorders relative to reality — which is the normal case for buffered mobile clients.
- Defining freshness SLOs on ingestion time buys a commitment you can actually meet and costs consumers the thing they often care about, which is completeness in event time. Both numbers need to be published; publishing only one guarantees an argument (Pipeline SLOs).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALEvery system that accepts data from elsewhere has a moment of acceptance worth recording, whether that is a broker append, an object landing in a bucket or a file arriving on a share. The argument for stamping it does not depend on streaming at all.
- BROKER-SPECIFICKafka stores a per-record timestamp that is either the producer's create time or the broker's log-append time depending on a per-topic configuration, so the same field means different things on different topics of one cluster; Kinesis records carry a server-side approximate arrival time; Pub/Sub carries a publish time assigned by the service. Ordering and monotonicity guarantees around these differ, so a rule learned on one does not transfer.
- SIMPLIFIEDDescribed as one stamping point. Real pipelines often have several boundaries — a collector, a broker, an object-storage landing — each of which could reasonably be called ingestion; the useful discipline is to name which boundary the field means and keep it fixed.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why the platform's clock is itself only a local claim, and what it means for two brokers to disagree about the order of two records they each accepted.