The question this answers
Two machines timestamped two events. Can I use those timestamps to order them?
None across machines. A physical clock reading orders events on the machine that produced it, and only if that machine has not stepped its clock in between. Across two machines, t_A < t_B implies nothing at all about which event occurred first. Under synchronisation the only honest claim is a bounded one: if the two readings differ by more than the worst-case clock error ε, and you can actually bound ε, the earlier reading probably came first.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows the current value of its own counter, and — if it runs an NTP client — the size and direction of the last correction its daemon applied and a server-reported round-trip estimate. It does not know the true time, does not know any other machine's offset, and cannot detect that its own clock is wrong. A machine with a clock two seconds fast reports the same confident timestamp as a machine with a correct one.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
12:00:01 and 12:00:03
Two services log to the same aggregator. Service A records user.email.changed at 12:00:01.400. Service B records email.sent at 12:00:03.100. The obvious reading is that the change happened, then the mail went out with the new address. That reading may be exactly backwards.
Both timestamps are true statements about the *local counter* of the machine that wrote them. Neither is a statement about a shared timeline, because there is no shared timeline — see There Is No Global Clock. A wall-clock reading is the output of a crystal oscillator counting at a nominal rate, plus whatever corrections a daemon has applied to it, on that machine, without reference to any other.
The strong form of the point: even if the two clocks were *perfectly* synchronised, the timestamps would still not order the events, because the events were recorded at different points inside their own operations. The ordering question is about causality, and a timestamp is not evidence of causality.
What a physical clock actually is
A quartz oscillator on the board ticks at a nominal frequency; a counter accumulates ticks; the OS turns that count into a date. The oscillator is not exact. Its error is quoted in parts per million: a mundane server crystal drifts on the order of tens of ppm, which is seconds per day, and the rate itself moves with temperature — the same box drifts differently under load than at idle (Computer Architecture owns the oscillator and the clock domain itself).
NTP exists to correct this. A client asks a server for the time, halves the round trip to estimate one-way delay, and computes an offset. That estimate is only as good as the *symmetry* assumption: if the outbound and return paths have different delays — a congested uplink, a busy virtualised NIC, a middlebox — the estimate is wrong by half the asymmetry, and NTP cannot detect the asymmetry from inside.
The daemon then applies the correction one of two ways, and the difference matters enormously to your code. Slewing speeds up or slows down the clock slightly until it converges — time stays monotonic, but a "second" is briefly not a second. Stepping jumps the clock to the new value — which can move it *backwards*. Most daemons slew small errors and step large ones, so the dangerous case is exactly the one that follows a long disconnection or a VM resume.
- Drift — the clock runs at the wrong rate. Continuous, gradual, always present.
- Offset — the clock currently reads the wrong value. What NTP measures and corrects.
- Skew — the difference between two clocks at the same instant. What actually breaks distributed logic; see Clock Skew: The Gap You Cannot Measure From Inside.
- Step — a discontinuous correction. The clock may go backwards, and durations computed across it become negative.
- A VM that is paused, migrated or restored from a snapshot resumes with a clock that is simply wrong, and it is wrong from the first instruction, before any daemon runs.
What a timestamp is legitimate evidence of
Timestamps are not useless. They are useless for *ordering across machines*, which is a much narrower claim. They remain good for the things they actually support.
Use a wall-clock timestamp when you want a human-meaningful label ("this happened around lunchtime on Tuesday"), when you need retention or expiry against an external calendar, or when you are correlating with the outside world — a customer's complaint, a third party's invoice. Do not use it as a tiebreaker in a merge, as a sequence number, or as a proof that one write superseded another. That last one is Last Write Wins Is Data Loss You Chose by Default, and it is the most expensive misuse of a timestamp in the field.
| Use | Sound? | Why |
|---|---|---|
| Display "3 minutes ago" to a usertypical | Yes | Approximate by construction; a second of error is invisible. |
| Order two events on the same machineassumption | Usually | Sound *unless* the clock stepped between them — use a monotonic source instead: [[monotonic-vs-wall-clock]]. |
| Order two events on different machinesprotocol | No | The readings come from unrelated oscillators with unknown offsets. |
| Decide which of two concurrent writes winsprotocol | No | Silently discards the loser, and "later" is decided by whichever machine's clock is fastest. |
| Expire a cache entry or a leaseassumption | Risky | Two parties must agree the deadline passed; use a duration on a monotonic clock plus a fencing token: [[leases]], [[fencing-tokens]]. |
Bounded uncertainty: the honest version
There is a way to make physical time usable for ordering, and it is instructive because of what it costs. Instead of returning a point, the clock API returns an interval: "the true time is somewhere in [earliest, latest]". Google's TrueTime does this with GPS receivers and atomic clocks in every datacentre, and Spanner uses it to give externally consistent transactions.
The trick is not the accuracy. It is that the uncertainty is exposed rather than hidden, so the system can wait it out. To commit a transaction at timestamp T, Spanner waits until T is definitely in the past everywhere — a *commit wait* of roughly the uncertainty width. Tighter clocks do not remove the wait; they shorten it. The correctness comes from waiting, and the hardware only buys latency back.
That is the trade worth internalising: you can buy ordering from physical clocks, and the price is a deliberate delay proportional to how badly you know the time. Cheap clocks are not incorrect here, only slow. A system that reads a plain now() and orders by it has not avoided the price — it has simply not paid it, and takes the incorrectness instead.
// what most runtimes give you — a point, with no error bar
Date.now() -> 1755993601400
// what a bounded clock gives you
now() -> { earliest: ...601380, latest: ...601420 } // ε = 40ms
// and the rule that makes it usable:
// commit at T, then sleep until now().earliest > T
// cost: one ε per externally-ordered commitKey points
- A wall-clock timestamp is a reading of one machine's oscillator, not a position on a shared timeline.
- Comparing timestamps from two machines is never a sound ordering, regardless of how well they are synchronised.
- NTP estimates offset from a round trip and assumes the path is symmetric — an assumption it cannot verify.
- Corrections are applied by slewing (time stretches) or stepping (time may jump backwards). Both break naive duration arithmetic.
- Physical time can order events only if the uncertainty is bounded *and* the system waits that bound out. Exposing ε is the mechanism; the hardware only makes ε smaller.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A crystal oscillator ticks at a nominal rate with an error of tens of ppm, varying with temperature and load.
- • The kernel accumulates ticks into a counter and maps it onto a calendar to produce wall-clock time.
- • An NTP client polls one or more servers, estimates one-way delay as half the round trip, and derives an offset.
- • The daemon applies the offset by slewing (adjusting the tick rate) for small errors or stepping (jumping the value) for large ones.
- • Any process reading the clock between those corrections sees a value whose error it cannot observe or bound.
- • The NTP path is asymmetric, so the computed offset is wrong by half the asymmetry and stays wrong.
- • The NTP server itself is wrong, or the pool disagrees, and the client picks a bad source.
- • The daemon is not running at all — a container image that never started one, a host that lost its upstream.
- • A VM is paused, migrated or restored, and resumes with a clock that is arbitrarily stale.
- • A large correction steps the clock backwards, so two events recorded in causal order carry decreasing timestamps.
- • Reordered logs: the operator opens an aggregated timeline and sees the response logged before the request that caused it. Nothing is broken in either service; the two hosts disagree by 300 ms.
- • Silent lost update: two replicas take a write, the merge picks the higher timestamp, and the write from the correct machine is discarded because the other machine's clock ran fast. The operator observes a user's edit vanishing with no error, no log line, and no conflict recorded anywhere.
- • Mass authentication failure after a clock step: a host jumps forward past a batch of token expiry times and starts rejecting every request as expired. The operator sees a 401 cliff on exactly one host in a fleet of forty.
- • Negative durations in metrics: a p99 histogram acquires a bucket at or below zero after an NTP step, and the aggregate latency chart drops to an impossible value for one scrape interval.
- • Scheduled work fires twice or not at all: a cron-like scheduler steps its clock backwards across a boundary and re-runs the same hourly job, or steps forward and skips one. The operator sees duplicate or missing batch output with no failed run.
- • No coordination is required to *read* a clock — which is exactly why clock reads are so attractive and so dangerous. They look free because they are free.
- • Making physical time usable for ordering requires either coordination (agree an order explicitly — see Total Order Broadcast Is Consensus Wearing a Different Hat) or waiting (pay one uncertainty bound per ordered operation).
- • NTP itself is coordination, but weak coordination: it converges clocks without ever agreeing an order, and it offers no bound you can safely program against.
- • Timestamps continue to be produced, confidently, with no indication that they are wrong. This is the defining property of the failure: it is silent.
- • Per-machine ordering of events survives a slew but not a step.
- • Anything built on causality rather than clocks — Lamport Clocks: Consistent With Causality, Blind to Concurrency, Vector Clocks: Buying Concurrency Detection at O(N) — is entirely unaffected by clock error. That is the reason those exist.
- • Detect: alert on the NTP offset and on the daemon's reachability, per host, not as a fleet average — the failure is always one host.
- • Contain: take a host with a large offset out of rotation rather than letting it serve writes; a fast clock on a write path is a data-loss source.
- • Recover: prefer slewing back into line where the offset is small enough to allow it, so nothing observes a backwards jump.
- • Reconcile: for data already merged by timestamp, there is usually no recovery — the losing version was discarded. This is why the fix is to stop ordering by timestamps, not to synchronise harder.
- • Verify: replay a known causal sequence across hosts and check the aggregated order matches the real one.
- • Per-host NTP offset and jitter, plotted as a distribution across the fleet. The outlier host is the story; the median is not.
- • Count of clock *steps* applied, as an event, not a gauge. A step is worth waking someone for on a write path.
- • Occurrences of negative or absurd computed durations in application metrics — an excellent free detector for clock steps you are not otherwise monitoring.
- • For logs: the rate of parent-after-child orderings in a trace, which is a direct measure of cross-host skew showing up in your data.
- • Human-facing labels, retention windows, billing periods, and anything anchored to an external calendar that people share.
- • Coarse correlation across systems where being a second out changes nothing.
- • As a *bounded* source with an explicit ε, when you own the clock infrastructure and are prepared to pay the wait.
- • Any tiebreak, merge, sequence number, or "who wrote last" decision. The failure is silent data loss.
- • Lease and lock expiry across machines, where two parties must agree that a deadline has passed.
- • Any measurement of elapsed time — use a monotonic source: Never Measure a Duration With the Wall Clock.
- • Order by causality instead of by time: Lamport Clocks: Consistent With Causality, Blind to Concurrency gives you a cheap order consistent with causality; Vector Clocks: Buying Concurrency Detection at O(N) additionally tells you when two events are genuinely concurrent.
- • Have one machine assign the numbers. A single sequencer sidesteps the whole problem, at the cost of a coordination point — see Distributed Uniqueness: One Name, Many Shards.
- • Store *both* versions instead of picking one, and resolve at read time with a real merge rule: Only the Application Knows What the Merge Means.
- • Use a bounded-uncertainty clock and wait out ε, if your infrastructure provides one and you can afford the added commit latency.
- • For duration, use a monotonic counter; for identity, use a UUID; for order, use a version. None of these needs the wall clock at all.
Two oscillators, drifting apart
host-a drift -12 ppm error -32.4 ms last step: none host-b drift +38 ppm error -1097.4 ms last step: -1.2 s event on host-a at true t -> stamped -32.4 ms off event on host-b at true t + 3 ms -> stamped -1097.4 ms off comparison of the two stamps: WRONG — the later event carries the earlier timestamp
What people believe, and what is true
We run NTP, so our clocks are synchronised.
NTP bounds the *typical* offset under good conditions. It offers no guarantee you can program against, does not detect asymmetric paths, and does nothing at all on the host where the daemon is not running.
The clocks are within a few milliseconds, so ordering by timestamp is fine.
Events that matter are often milliseconds apart — that is exactly the regime where the comparison is a coin flip. And "a few milliseconds" is a claim about the median host on a good day.
UTC is a monotonically increasing number of seconds.
UTC includes leap seconds. Depending on the platform and the smearing policy, the same second can repeat or a second can be stretched.
The database's timestamp is authoritative.
It is authoritative for that server. In a multi-primary or multi-region deployment there are several such servers, each authoritative and each different.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A timestamp tells you what one machine's clock said. It does not tell you when something happened relative to another machine. Never order across machines by timestamp.
Practical
Keep timestamps for display, retention and external correlation. For ordering, use a version or a sequence; for duration, use a monotonic clock; for merge decisions, use a causality mechanism. Monitor NTP offset per host and alert on clock steps as events.
Advanced
If you need physical time to order, use a clock that returns an interval and wait out the width before you claim a timestamp is in the past. This is Spanner's commit wait. Correctness comes from the wait; better hardware only shrinks it. The general lesson: uncertainty you expose can be paid for, uncertainty you hide becomes silent corruption.
Apply it
- 🔧 Find every place in your codebase that compares a timestamp produced on one machine with one produced on another. Classify each as display, ordering, or merge.
- 🔧 Write a test that steps the system clock backwards mid-request and assert nothing produces a negative duration or a duplicated scheduled run.
- ⚡ A support ticket says a user's profile edit "undid itself". Two app servers took writes seconds apart and the merge kept the one with the larger timestamp. Which host would you look at first, and what would you measure?
- 💬 Service A logs at 12:00:01 and service B logs at 12:00:03. Can you conclude A's event came first? Justify your answer precisely.
- 💬 What does NTP actually measure, and what assumption does that measurement rest on?
- 💬 Spanner uses atomic clocks. What does the hardware actually buy, given that the correctness argument is a wait?