OperationsGENERALCLOUD-SPECIFICSIMPLIFIED

Clock Synchronisation

Machines need reasonably synchronised clocks for logs, certificates, tokens and scheduling — reasonably, because perfect synchronisation is not available.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

How closely do production clocks need to agree, and what breaks when they drift?

The problem

Several production mechanisms compare a local clock reading against something produced elsewhere — a certificate validity window, a token expiry, a log timestamp — and quietly assume the two clocks agree.

What teams do first

Machines have clocks and the cloud provider keeps them right. Clock drift is a distributed systems curiosity, not an operational concern.

How it breaks

A machine whose clock is behind rejects a freshly issued certificate as not yet valid, and one whose clock is ahead rejects a valid one as expired (Certificates as an Operational Object).

How it breaks in production

A machine whose clock is behind rejects a freshly issued certificate as not yet valid, and one whose clock is ahead rejects a valid one as expired (Certificates as an Operational Object).
Signed tokens with short lifetimes fail validation between two hosts that disagree, producing authentication failures that look like an identity provider problem.
Logs from a drifted host interleave incorrectly, so an incident reconstruction shows an effect before its cause (The Debugging Timeline).
A scheduler on a drifted host triggers early or late relative to everything else, which matters when work is coordinated across machines.
Drift is gradual and usually silent: nothing alerts, and the symptom appears as an unrelated failure somewhere else.
A large step correction — the clock jumping to catch up — can be worse than the drift, because code that measured elapsed time using the wall clock sees time move backwards.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

Each machine has a local oscillator that is imperfect, so its clock drifts relative to a reference. A synchronisation daemon corrects it continuously against upstream time sources.
Correction is normally applied by slewing — speeding the clock up or slowing it down slightly until it agrees — rather than by stepping, because a step can move time backwards and break anything measuring an interval.
This is why elapsed time should be measured with a monotonic clock, which only moves forward and is unaffected by corrections, while wall-clock time is used only for recording when something happened.
The realistic goal is a bounded disagreement, not equality. Synchronisation gives you clocks that agree closely enough for the mechanisms above, and no protocol gives you certainty that two clocks read the same value at the same instant.
That uncertainty is exactly why timestamps cannot establish ordering between events on different machines. Deciding what happened before what is a separate problem with separate tools.
Practically, the tolerances differ by mechanism: certificate validity and token expiry commonly allow a small skew explicitly, while log correlation degrades gradually rather than failing outright.

What depends on clocks agreeing

GENERALTolerances differ by mechanism and implementation: certificate and token validation commonly permit a small explicit skew, while log correlation degrades continuously rather than failing at a threshold. Check the tolerance of the specific mechanism rather than assuming a single number applies.

Each row compares a local clock reading against something produced on another machine. That comparison is the dependency, and its tolerance differs per mechanism.

Mechanism	What it compares	Symptom when clocks disagree
TLS certificate validity	Local clock against the certificate's validity window	Connections refused as not-yet-valid or expired (Certificates as an Operational Object)
Signed tokens	Local clock against issue and expiry claims	Authentication fails between two healthy services
Log correlation	Timestamps from different hosts	Effects appear before causes in an incident timeline
Scheduling	Local clock against an intended trigger time	Coordinated work triggers early or late relative to other hosts
Metrics and traces	Timestamps across a request path	Spans that appear to start before their parent
Rate limiting and caching windows	Local clock against a window boundary	Windows that do not line up across instances of the same service

Two clocks, two jobs

Almost every clock bug in application code comes from using one clock for the other's job. The distinction is simple and rarely taught alongside the API that exposes both.

Wall clock versus monotonic clock

Wall clock for a duration

start = wall_clock_now()
... do work ...
elapsed = wall_clock_now() - start

# a synchronisation step during the work can make
# elapsed negative, or inflate it by the correction;
# a timeout built this way can fire instantly or never

Monotonic for elapsed, wall for instants

start = monotonic_now()          # only ever moves forward
... do work ...
elapsed = monotonic_now() - start  # safe across corrections

log.info("finished",
         at = wall_clock_now_utc(),  # when it happened
         took = elapsed)             # how long it took

The monotonic clock is not affected by synchronisation, so durations, timeouts, retries and rate limits stay correct while the wall clock is being corrected. The wall clock is the only one that can say *when* something happened in a way another machine can interpret — so it belongs in the log line and nowhere near the subtraction.

Drift as an operational signal

Clock offset is one of the cheapest metrics to collect and one of the most useful during a confusing incident, because it turns a mysterious authentication failure into a one-line diagnosis.

Add host clock offset to the operator dashboard next to authentication and TLS error rates; the correlation does the diagnosis for you.
Check offset early when a failure involves validity, expiry or signatures. It is a cheap check with a high hit rate for a confusing symptom.
Ordering across machines is not a clock problem you can solve by synchronising harder.

Clock-related incidents

Trigger	Symptom	Cause	Response
New host joins the fleet	Token validation fails only on that host	Serving traffic before synchronisation converged	Gate readiness on clock synchronisation at startup (Probes: Readiness, Liveness and Startup)
Network policy change	Gradual, fleet-wide drift over days	Time sources became unreachable and nothing alerted	Monitor daemon health and offset as first-class signals
Large step correction applied	Negative durations; timeouts firing immediately	Elapsed time measured on the wall clock	Use the monotonic clock for durations; prefer slewing
Certificate renewed	A subset of clients reject the new certificate	Those clients' clocks are behind the validity start	Check client offset; allow a small skew on validity start (Renewal: Automating the Thing That Expires)
Trace collected across services	Child span starts before its parent	Host clocks disagree by more than the span duration	Read trace structure from the parent-child relationship, not from timestamps
Rate limit windows across instances	Effective limit is higher than configured	Per-instance windows are offset from one another	Use a shared counter for limits that must be global, not per-instance wall-clock windows

How to do it properly

Most important first.

Run a synchronisation daemon on every host, including containers' underlying nodes, and treat the underlying host as the source of truth rather than the container.
Monitor clock offset per host and alert on it — this is a cheap signal that prevents a class of confusing failures (Using Observability, Not Building It).
Use the monotonic clock for durations, timeouts and rate limiting; use the wall clock only for recording instants (Production Time Is UTC).
Prefer slewing over stepping in normal operation, and treat a large step as an event worth logging.
Allow a small, explicit skew tolerance where a protocol supports it, rather than assuming exact agreement between issuer and verifier.
When a certificate or token failure appears, check clock offset early — it is a cheap check that resolves a confusing class of incident quickly (Production Debugging).
Do not build correctness on timestamp comparison across machines. If ordering matters, use a mechanism designed for ordering.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongOne zone

One testEveryone

What contains it

Per-host offset alerting and a startup gate contain it to the affected hosts, which is why the failure is usually partial and intermittent. A time source outage affecting a whole fleet is not contained by anything host-local.

What can go wrong

Failure modes, including of the mitigation

A host boots with a wrong clock and serves traffic before synchronisation completes, failing every token validation until it catches up.
Time sources are unreachable due to a network policy, so drift proceeds unnoticed with no alert.
A large step correction moves time backwards and a service measuring elapsed time with the wall clock computes a negative duration.
Skew tolerance is set generously to make an intermittent failure go away, which widens a security window at the same time.

Misreads this invites

"Synchronised clocks mean identical clocks." They mean bounded disagreement. No synchronisation protocol delivers exact agreement, and building on the assumption that it does produces subtle bugs.
"We can order events across services by timestamp." You cannot, and this is the misreading with the largest consequences — ordering across machines needs a mechanism designed for it, not a clock.
"The cloud handles time." Providers offer good time sources; whether your hosts reach them, and whether the daemon is healthy, is still yours to verify.
"Use the wall clock for timeouts." A wall clock can move backwards during a correction. Durations belong on the monotonic clock.
"Clock skew is a rare, exotic failure." It is a routine cause of certificate and token errors, and it is one of the cheapest things to check.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

Clock offset per host exported as a metric, with an alert threshold and a visible history.
Synchronisation daemon health monitored as a service, not assumed because it is installed.
Durations in application code measured with a monotonic source, verifiable by reading the code.
An incident in which a certificate or token failure was diagnosed by checking offset — a sign the check is part of the routine.

How you get back

Correcting a badly wrong clock is the fix and is itself disruptive: prefer slewing, and if a step is unavoidable, do it with the host out of rotation.
Any state derived from wrong timestamps — records written with a bad clock — may need a data correction, and that is a recovery operation rather than a configuration change (Partial and Logical Data Recovery).
Widening skew tolerance is a fast mitigation and a poor fix; record it as temporary with a follow-up, because it weakens a validity check (Action Items That Change the System).

What to automate, and what stays human

Automate: synchronisation daemon installation and health checks, offset metrics and alerting, and a startup gate that refuses to serve traffic until the clock is synchronised.
Automate the diagnostic — surface host clock offset on the operator dashboard next to certificate and authentication error rates, so the correlation is visible without anyone thinking of it.
Keep human: choosing skew tolerances, since each one is a trade between reliability and the strictness of a security check.

What this costs

Tighter synchronisation requires reachable, trusted time sources and monitoring; looser tolerance is easier and widens validity windows.
Slewing avoids backwards jumps and takes longer to converge, so a badly wrong clock stays wrong for a while.
Gating startup on synchronisation prevents a class of failure and adds a startup dependency that can itself delay a recovery (Probes: Readiness, Liveness and Startup).

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

GENERALEvery machine needs a synchronisation daemon and every fleet needs offset visibility. This holds on-premise and in every cloud; only the default time source differs.
CLOUD-SPECIFICProviders typically offer a local time service reachable from inside the network, which is more reliable than public internet sources and is usually the default in their base images. A custom image or a restrictive network policy can silently remove that default, which is the common way drift starts.
SIMPLIFIEDThis lesson deliberately stops at "reasonably synchronised, with bounded uncertainty". Bounding that uncertainty explicitly and using it for ordering or consistency decisions is a distributed systems topic and is not taught here.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Observability & Performanceincident-timeline

Securitycertificate-trust audit-logs

OS & Networkingtls-handshake why-tls

Domains that do not exist yet

— Distributed Systems — logical clocks, causality and happened-before: the mechanisms that establish ordering between events on different machines, which physical clock synchronisation cannot do.