The question this answers
How does authority get revoked from a node that has stopped responding — and what does the expiry actually assume?
A lease guarantees that the issuer will not grant to another party before the lease expires by the issuer’s clock. It does not guarantee that the holder stops acting at that moment: the holder’s clock differs, its process may be paused, and it may not have learned anything. So a lease bounds the issuer’s behaviour precisely and the holder’s behaviour only under an assumption about clock drift and scheduling.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A holder knows when it *received* the grant by its own clock and how long the lease was said to last. It does not know the issuer’s current time, its own clock’s drift, or whether it has been descheduled since. So "my lease is still valid" is an inference from a local clock about a remote decision — two sources of error stacked. The safe form is to treat the lease as expiring earlier than stated, and to treat any renewal failure as immediate loss of authority.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Why authority must expire
The founding problem is that a holder can die without telling anyone. If authority is granted indefinitely, a crashed leader holds leadership forever and nothing can proceed — the system has traded a liveness failure for a safety guarantee, which is rarely the trade you wanted.
A lease attaches a deadline: authority is valid until time T unless renewed. If the holder is alive it renews well before T and nothing changes. If it dies, T passes and the issuer is free to grant to someone else, with no need to detect the death — the absence of renewal *is* the detection, and it requires no failure detector to be accurate.
This is the same mechanism as From Alive-or-Dead to a Suspicion Level, viewed from the direction of authority rather than membership. A heartbeat says "I am alive"; a lease says "my authority lapses unless I keep saying so". The second framing is safer, because the default on silence is to lose authority rather than to keep it.
The clock assumption, stated honestly
The issuer decides expiry by its clock; the holder decides whether it still holds by its clock. For the two to agree, their clocks must not drift far apart over the lease duration — and, more importantly, the holder must actually be executing to notice its clock advancing.
That second requirement is the one that fails. A process paused by garbage collection, a suspended virtual machine, or a container starved of CPU experiences no time at all. It resumes and reads a clock that has jumped forward — if it thinks to read it. Very often the code path is "I checked when I acquired; I am now writing", and the check never happens again.
Two disciplines follow. Use a monotonic clock for lease arithmetic, never wall time, because wall time can step backwards under NTP correction and hand you authority you do not have — see Never Measure a Duration With the Wall Clock. And apply a safety margin: treat the lease as expiring meaningfully earlier than granted, so the holder stops before the issuer is entitled to reassign. The margin covers drift and the delay between deciding to act and the effect landing.
Renewal, and what to do when it fails
A holder doing work longer than one lease period must renew: a periodic request that extends the deadline. Renewal is where two design decisions matter more than they appear.
First, renew early — at a third of the lease period is a common choice — so that a couple of lost renewals do not cost you authority. Renewal traffic is cheap; losing authority mid-work is not.
Second, and far more important: decide in advance what the holder does when renewal fails. The correct answer is almost always to stop immediately, before the lease expires, and abandon in-flight work. The tempting answer — "finish this operation, it will only take a moment" — is precisely the behaviour that produces two actors, because by the time the operation completes the issuer may have reassigned.
A holder that cannot renew has lost authority whether or not it has been told. The only safe posture is to act as though it has, and to make sure the resource will reject it if it is wrong.
1lease = issuer.acquire("shard-7", ttl=30s) # returns token + ttl2deadline = monotonic_now() + lease.ttl - SAFETY_MARGIN # never wall clock3token = lease.token4 5loop:6 if monotonic_now() > deadline:7 stop_immediately() # do NOT finish the current operation8 break9 10 do_a_unit_of_work(token) # every effect carries the token11 12 if time_to_renew(): # at ~1/3 of the ttl13 r = issuer.renew("shard-7", token)14 if r.ok: deadline = monotonic_now() + r.ttl - SAFETY_MARGIN15 else: stop_immediately(); break # renewal failure == authority lostWhere leases buy something a lock cannot
The most valuable use of a lease is not exclusion at all — it is making a read safe without a round trip. A leader holding a lease knows that no other leader can have been granted authority before the lease expires, so within that window it may answer reads from local state without confirming with a quorum. This is how leader leases turn linearizable reads from a round trip into a local operation, and it is a large performance win.
The catch is the same one throughout: that guarantee holds only under a bounded-drift assumption. If the leader’s clock runs slow relative to the issuer’s, it can believe its lease is live after another leader has been granted one, and serve a stale read while claiming freshness. Implementations that take this seriously either use a conservative margin large enough to swamp plausible drift, or use a tightly-bounded clock service and state the bound explicitly.
The other genuine use is graceful revocation: a lease can be allowed to lapse to remove authority from a node that has become unreachable, without any need to reach it. Revocation that requires contacting the revokee cannot work in exactly the case you need it.
The honest summary of what a lease is
A lease converts "who has authority?" — a question requiring coordination — into "is my local clock past a number?", a question requiring none. That is an excellent trade for availability and latency, and it is why leases appear in every mature distributed system.
What it does not do is make the holder’s belief correct. It bounds when the *issuer* will reassign, and everything else rests on assumptions about clocks and scheduling that the holder cannot verify. For efficiency purposes those assumptions are fine. For correctness, pair the lease with a fencing token so that being wrong about the clock produces a rejected write rather than a corrupted resource — which is The Stale Lock Holder: A Paused Process Does Not Know It Was Paused and Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
Key points
- Authority must expire, or a dead holder holds it forever and no failure detector can help.
- A lease bounds the issuer’s behaviour precisely and the holder’s only under a clock-drift assumption.
- Use a monotonic clock for lease arithmetic and subtract a safety margin covering drift and in-flight delay.
- Renew early, and treat any renewal failure as immediate loss of authority — stop, do not finish.
- A paused process experiences no time, so no amount of clock discipline covers it; only fencing does.
- Leases enable local linearizable reads and revocation without contacting the revokee — their two genuine wins.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A client requests authority; the issuer grants it with a duration and a monotonically increasing token.
- • The holder computes a local deadline from a monotonic clock, minus a safety margin.
- • The holder performs work, attaching the token to every effect.
- • Well before the deadline the holder renews; a successful renewal extends the local deadline.
- • If renewal fails or the deadline passes, the holder stops immediately.
- • The issuer, seeing no renewal, allows the lease to lapse and may grant to another party with a higher token.
- • The grant reply is delayed, so the holder’s effective window is shorter than it believes.
- • Clocks drift, so issuer and holder disagree about the expiry moment.
- • The holder pauses and observes no elapsed time at all.
- • Renewal requests are lost while the holder is healthy, causing an unnecessary loss of authority.
- • A renewal succeeds at the issuer but the reply is lost, so the holder stops despite being entitled to continue.
- • Wall-clock time steps backwards under NTP correction, extending a lease that should have ended.
- • Two active owners after a pause: the operator sees duplicated effects with both processes logging normal operation, and an expiry event in the issuer’s log between them.
- • Lease flapping under load: renewals queue behind work on a saturated node and arrive late, so authority moves repeatedly. The operator sees ownership changing every few seconds and throughput collapsing, on a node that is up the whole time.
- • Stale local reads: a leader with a slow clock serves reads under a lease it no longer holds. The operator sees clients reading values older than writes they have already been told succeeded — a linearizability violation with no error anywhere.
- • Authority lost on a network blip: renewal fails for two intervals and a healthy holder stops work. The operator sees an unnecessary failover and a work gap equal to the lease period.
- • Lease extended by a backwards clock step: wall-clock arithmetic plus an NTP correction leaves a holder acting past a genuine expiry. The operator sees a double-owner window correlated with a time-sync event.
- • Coordination happens at grant and at each renewal — periodic rather than per-operation, which is the whole point.
- • Between renewals the holder acts with no coordination at all, at local speed.
- • The lease period is the tuning dial between coordination traffic and failover time: shorter means more traffic and faster recovery, longer means the opposite.
- • If the holder dies, authority lapses automatically after at most one lease period, with no detection required.
- • If the issuer is unreachable, the holder loses authority at its deadline and stops — fail-closed by construction.
- • If the holder is merely paused, the lease does not protect the resource; only a token check does.
- • Detect: track renewal failures and expiries-during-work separately; the latter is the dangerous one.
- • Contain: enforce the local deadline check before every effect, not only at the top of a loop, and fence the resource.
- • Recover: after losing authority, re-acquire and obtain a fresh, higher token before resuming any work.
- • Reconcile: check for duplicate effects across each expiry boundary — that is the only window in which they occur.
- • Verify: run a pause test — suspend a holder past its lease, resume, and confirm its writes are rejected.
- • Renewal latency and failure rate per lease.
- • Count of leases that expired while the holder was still working.
- • Ownership-change frequency, which should be near zero in steady state.
- • Measured clock offset between issuer and holders, against the safety margin you chose.
- • Rejected-token counts at the protected resource.
- • Whenever authority must be revocable from a node that may be unreachable.
- • Whenever coordination should be amortised: pay once per lease period instead of once per operation.
- • For leader leases enabling local reads, which is one of the largest available latency wins in a consensus system.
- • When the lease period is short relative to the work, so renewal becomes a dominant failure source.
- • When the environment has unbounded pauses — heavily oversubscribed hosts, aggressive GC — making the clock assumption unsupportable.
- • When used as a correctness mechanism without fencing, which is the same misuse as with locks.
- • Explicit revocation with acknowledgement, which is safer when it works and impossible when the holder is unreachable — that is, exactly when you need it.
- • Fencing tokens without a time bound: authority is not time-limited, but a stale actor is rejected anyway. Safer, and gives up automatic lapse.
- • Session-based ownership tied to a connection (ZooKeeper ephemeral nodes), where authority ends when the session does — still a lease, with the timeout managed by the service. See Coordination Services: The Primitives, Not the Product.
- • Per-operation confirmation with the issuer: maximum safety, maximum coupling, and no clock assumption at all.
Leases: authority with an expiry date
What people believe, and what is true
When the lease expires, the holder stops.
The issuer becomes free to reassign. Whether the holder stops depends on its clock and on whether it is running at all.
Longer leases are safer.
They reduce renewal failures and lengthen every recovery. The unsafe overlap window is unchanged.
Wall-clock time is fine for lease arithmetic.
It can step backwards under NTP correction and silently extend a lease that should have ended. Use a monotonic clock.
If renewal fails I can finish the current operation.
That is exactly the window in which a second holder starts. Stop immediately and abandon in-flight work.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A lease is authority with an expiry, renewed while the holder is alive. If the holder dies, authority lapses on its own — no detection needed. If the holder pauses, it may act past expiry, which is why the resource should also check a token.
Practical
Compute deadlines from a monotonic clock, subtract a margin, renew at a third of the period, and stop immediately on renewal failure. Monitor expiries that happen during work. Never treat the lease alone as a correctness mechanism.
Advanced
A lease trades a coordination assumption for a timing assumption, which is usually a good trade because timing assumptions can be made conservative with a margin while coordination cannot be made cheap. The residual risk is unbounded pauses, against which no margin is sufficient. Leader leases exploit the same trade for reads: local reads are safe within the lease window under bounded drift, which is why systems offering them must publish a drift bound and degrade when it is exceeded.
Apply it
- 🔧 Derive a lease TTL and safety margin from measured p99 work duration, measured clock offset and observed pause durations.
- 🔧 Explain how a leader lease makes a linearizable read local, and state precisely the assumption that makes it correct.
- 💬 Why does authority in a distributed system usually need an expiry?
- 💬 What exactly does a lease guarantee, and what does it only guarantee under assumptions?
- 💬 Why must lease arithmetic use a monotonic clock?
- 💬 Your renewal fails once. What should the holder do, and why not finish the current operation?