The question this answers
Which clock do I read to measure how long something took?
A monotonic clock guarantees non-decreasing readings within a single process on a single machine, with a resolution the platform documents. It guarantees nothing about calendar time, is not comparable across machines, and on most platforms is not comparable across reboots. A wall clock guarantees the opposite: a shared calendar meaning, and no monotonicity at all.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A process knows the difference between two of its own monotonic readings, and that difference is a sound measurement of elapsed time on that machine. It does not know whether the wall clock moved between those readings, how far, or in which direction — nothing in the wall-clock API reports that a step occurred.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The bug, in four lines
This is the shape it always takes. Someone writes a timeout, a retry backoff, a rate limiter, a cache TTL, or a latency metric, and reaches for the obvious API.
Ninety-nine point nine percent of the time this is correct. The remaining fraction is an NTP step, a leap-second adjustment, an administrator running date, or a VM resume — and in that fraction, elapsed is wrong by the size of the step. If the step was backwards, elapsed is negative, and the loop below waits forever.
The failure has a particularly nasty signature: it is unreproducible, it correlates with unrelated infrastructure events, and it produces impossible numbers rather than errors. Engineers spend days on it because "the code obviously works".
1// WRONG — Date.now() is the wall clock. It can jump forwards or backwards.2const start = Date.now()3await doWork()4const elapsed = Date.now() - start // can be negative, or off by the step size5 6// WRONG in the worst way — a deadline loop on the wall clock7while (Date.now() - start < timeoutMs) { ... } // clock steps back => loop never exits8 9// RIGHT — a monotonic source. Never jumps, never goes backwards.10const start = performance.now() // browser/Node; process.hrtime.bigint() also fine11await doWork()12const elapsed = performance.now() - start // a real duration13 14// The rule, stated once:15// Wall clock -> "what time is it?" (display, retention, external correlation)16// Monotonic -> "how long was that?" (timeouts, backoff, TTLs, metrics, rate limits)Why there are two clocks at all
They are not redundant; they answer genuinely different questions and cannot be the same clock. The wall clock must track an external standard, which means it must be *correctable* — and correction means discontinuity. The monotonic clock must never go backwards, which means it must never be corrected to match anything, which means it cannot carry a calendar meaning.
You cannot have both properties in one clock. That is the entire reason the operating system exposes two, and the reason every mature runtime has a separate API for each. A language that gives you only now() has made the choice for you, badly.
The monotonic clock typically counts from an arbitrary origin — boot, or process start — so an individual reading is meaningless. Only *differences* mean anything. This is a feature: it makes the API hard to misuse for the calendar, in the same way that the wall clock ought to be hard to misuse for durations and unfortunately is not.
| Wall clock | Monotonic clock | |
|---|---|---|
| Answers | What date and time is it? | How much time has passed? |
| Can jump backwardsprotocol | Yes — NTP step, admin change, VM resume | No, by definition |
| Can be stretchedtypical | Yes — NTP slew, leap smearing | Usually no; rate is not disciplined |
| Comparable across machinesprotocol | Approximately, with unbounded error — see [[clock-skew]] | Not at all; different origins |
| Survives a reboottypical | Yes | Typically no — origin resets |
| Use for | Display, retention, billing periods, external correlation | Timeouts, backoff, TTLs, latency metrics, rate limits, leases |
Leap seconds, and the day the internet fell over
UTC is tied to the earth's rotation, which is irregular, so occasionally a leap second is inserted: the clock reads 23:59:60, or 23:59:59 twice, depending on the platform's coping strategy. A wall clock is therefore not a monotonically increasing count of seconds, and code that assumes it is will misbehave once every few years on a schedule you do not control.
The historically famous version of this took down large parts of several major services: a kernel path that assumed time never repeated ended up spinning, load spiked fleet-wide, and the correlation to the leap second was not obvious for hours because *nothing had been deployed*. The lesson people took away was "leap seconds are dangerous". The more useful lesson is that an assumption of monotonicity had been made about a clock that never promised it.
The mainstream mitigation today is smearing: spreading the extra second across many hours by slewing, so no repeat and no step ever occurs. Smearing means a "second" measured on the wall clock is not a second for the duration of the smear — which is fine for the calendar and wrong for durations, and is one more reason durations do not belong on that clock.
request_duration_ms bucket count <= 10 41,203 <= 50 18,884 <= 100 2,110 <= 1000 311 <= +Inf 12 ---- after a -1.4s NTP step at 03:11:07 ---- observed values: -1399, -1398, -1391, 3, 7, 11, ... p50 for the scrape window: -1391 ms An impossible number is the tell. Latency histograms and rate limiters are usually the first places a clock step becomes visible in application data.
Where this bites in distributed code specifically
Inside one process the fix is mechanical: use the monotonic API. The distributed version is subtler, because durations start crossing machines and the monotonic clock cannot cross with them — two machines' monotonic clocks have unrelated origins and are entirely incomparable.
So a deadline propagated across a call graph (Pass the Remaining Budget Down, Not a Fresh One) cannot be sent as "monotonic reading 918273". It has to be sent as a remaining duration — "you have 240 ms" — which each hop then measures against its own monotonic clock. Sending an absolute wall-clock deadline instead reintroduces skew into a path that had eliminated it, and does so invisibly.
The same reasoning applies to leases (Leases: Authority With an Expiry Date): a lease should be granted as a duration the holder measures locally and the grantor measures locally, with the grantor waiting longer than the holder before reassigning. And to retry backoff (Without Jitter, Every Client That Failed Together Retries Together): a backoff computed on the wall clock can become instantaneous or infinite after a step, which turns a healthy retry policy into a retry storm at the worst possible moment.
Key points
- Wall clock answers "what time is it"; monotonic answers "how long was that". They cannot be the same clock, because correctability and monotonicity are incompatible.
- Any duration, timeout, backoff, TTL, rate limit or latency metric computed from
Date.now()is a latent bug that fires during NTP steps, leap-second handling and VM resumes. - A negative or impossible duration in your metrics is the signature; the bug produces no errors.
- Monotonic clocks do not cross machine boundaries — different origins. Propagate deadlines as *remaining durations*, not absolute times.
- UTC is not a monotonically increasing count of seconds. Leap seconds and smearing both break that assumption, on a schedule you do not control.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The kernel maintains a wall-clock offset that NTP may slew or step, and a separate counter derived from a hardware source that only advances.
- • A wall-clock read returns the corrected calendar value; a monotonic read returns the raw advancing counter, scaled.
- • Subtracting two monotonic reads yields elapsed time on that machine, unaffected by any correction.
- • Subtracting two wall-clock reads yields elapsed time *plus* any correction applied in between, with no way to tell which is which.
- • Across machines, monotonic origins differ arbitrarily, so only durations — never readings — may be transmitted.
- • An NTP step lands between two wall-clock reads, corrupting the computed duration.
- • A leap second repeats or stretches a second, so a wall-clock duration is off by up to a second and code assuming strict increase misbehaves.
- • A VM suspends between reads; on some platforms the monotonic clock excludes suspend time and on others it includes it — the platform choice is not the one your code assumed.
- • A process caches a monotonic reading and compares it after a restart, where the origin has changed.
- • An absolute deadline computed on a fast machine is sent to a slow one and expires on arrival.
- • Hung request after a clock step: a deadline loop written against the wall clock never exits because time went backwards. The operator observes threads stuck with no error, no timeout metric, and a connection pool that drains to zero.
- • Latency dashboard shows negative or absurd values for exactly one scrape interval, then recovers — usually the first visible symptom of an NTP step nobody was monitoring.
- • Retry storm at the moment of a correction: a backoff computed on the wall clock evaluates to zero for every client at once, and a dependency that was merely slow becomes overloaded.
- • Rate limiter opens or closes fleet-wide: a window computed on the wall clock jumps, and either every request is admitted for a second or every request is rejected.
- • Caches expire en masse: absolute TTLs computed before a forward step are all in the past afterwards, producing a synchronised stampede against the origin.
- • None inside a process — this is the rare case in the domain where the correct answer requires no coordination at all, only the right local API.
- • Across machines, the coordination is in the *protocol*: agreeing to send remaining durations rather than absolute deadlines, so each hop can use its own local clock.
- • A lease is coordination expressed as two independent local timers plus a safety margin, which is deliberately cheaper than agreeing on a shared instant.
- • Monotonic-based durations remain correct through NTP steps, leap seconds and administrator clock changes.
- • They do not survive a reboot or a process restart, and they are meaningless if sent to another machine.
- • Under VM suspend, whether suspended time is counted is platform-specific — the guarantee is monotonicity, not that it measures physical time.
- • Detect: assert on impossible values — a duration below zero, a TTL longer than it could be — and count them as an explicit metric.
- • Contain: clamp computed durations at zero at the measurement site so one bad reading cannot poison an aggregate or an infinite loop.
- • Recover: restart processes holding cached absolute deadlines derived before a step, since those deadlines are simply wrong now.
- • Reconcile: recompute anything expiry-driven that fired incorrectly during the window — re-warm caches, re-issue tokens.
- • Verify: inject a clock step in a test environment and confirm no duration goes negative and no deadline loop hangs. This is a cheap, repeatable test almost nobody writes.
- • A counter of negative or clamped durations, per call site. Non-zero means somebody is measuring on the wrong clock.
- • Clock step events from the time daemon, correlated against latency and error charts — the correlation is the diagnosis.
- • Distribution of deadline-remaining values arriving at downstream services; a bimodal distribution with a cluster at zero indicates absolute deadlines crossing a skewed boundary.
- • Retry inter-arrival times, which collapse toward zero if backoff is computed on a wall clock that just moved.
- • Every timeout, retry backoff, latency measurement, rate-limit window, lease and cache TTL. There is no case where a duration is better measured on the wall clock.
- • Especially in long-lived processes, where the chance of experiencing a correction during the process lifetime approaches one.
- • The monotonic clock is wrong for anything that must mean the same thing to a human or another machine — do not persist it, log it as a date, or send it across a boundary.
- • Do not use it for scheduling against a calendar ("run at midnight"); that genuinely is a wall-clock question, with all the hazards that implies.
- • If your platform exposes only one clock, wrap it once behind a
now()for the calendar and anelapsed()for durations, and enforce the split in review — the discipline is the mechanism. - • For cross-machine deadlines, send remaining duration and let each hop time itself locally: Pass the Remaining Budget Down, Not a Fresh One, A Deadline Is Divided Across the Call Chain, Not Repeated at Every Hop.
- • For expiry that must be agreed by two parties, use a lease with independent local timers and an asymmetric margin: Leases: Authority With an Expiry Date.
- • For "has this been superseded", use a version rather than a time at all: Version Vectors: Making the Conflict Visible.
Never measure a duration with the wall clock
const t0 = Date.now() // wall clock: corrected without notice await handler(req) const took = Date.now() - t0 // took = -1358.0 ms ✗ negative — the clock moved backwards mid-request
observed values: -1399, -1398, -1391, 3, 7, 11, ...
p50 for the scrape window: -1391 ms
What people believe, and what is true
Date.now() is fine because clock steps are rare.
Rare, uncorrelated bugs are the expensive kind. This one fires during infrastructure incidents — precisely when you need your timeouts and backoff to behave.
Using UTC avoids the problem.
UTC is a wall clock. It is still stepped by NTP, and it still contains leap seconds. The time *zone* is unrelated to the monotonicity question.
The monotonic clock is more accurate.
It is not more accurate about the date — it has no opinion about the date. It is simply never corrected, which is what makes differences meaningful.
I can send a monotonic reading to another service and compare it there.
The origins are unrelated. The difference between two machines' monotonic clocks is an arbitrary number with no meaning at all.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Two clocks. The wall clock knows the date and can jump; the monotonic clock only counts forward and does not know the date. Durations go on the monotonic one, always.
Practical
Audit every subtraction of two wall-clock reads. Replace with a monotonic source. Clamp durations at zero and count the clamps. Propagate deadlines as remaining durations across services. Add a test that steps the clock backwards mid-request.
Advanced
The two properties are provably incompatible: a clock that tracks an external standard must be correctable, and correction is discontinuity. So the OS exposes both and pushes the choice to you. In distributed code the monotonic clock stops at the machine boundary, which is why deadline propagation is expressed in durations — it is the only representation that survives having no shared time base.
Apply it
- 🔧 Grep for
Date.now()/time.time()/System.currentTimeMillis()subtractions and classify each as a duration (bug) or a calendar use (fine). - 🔧 Write an integration test that steps the container clock backwards by two seconds during a request and assert no hang and no negative duration.
- ⚡ During a routine NTP correction, one service's connection pool exhausts and never recovers until restart. No errors are logged. Where do you look?
- ⚡ A rate limiter admits ten times the configured rate for exactly one second, once, and cannot be reproduced.
- 💬 Why does the operating system expose two clocks, and what would break if it exposed only one?
- 💬 Your latency dashboard shows a negative p50 for one minute. What happened, and what else should you check?
- 💬 How do you propagate a 500 ms deadline across four services with no shared clock?