The question this answers
When can I trust a distributed lock, and what is it actually protecting?
By itself: at most one holder as far as the lock service is concerned, which is a statement about the lock service’s records rather than about the world. It does not guarantee at most one *actor*, because a holder can lose the lock while continuing to act. Extended with a fencing token checked at the resource, the guarantee becomes "at most one actor can successfully affect the resource" — and only then is it usable for correctness.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A holder knows the lock service granted it the lock, and when. It does not know whether it still holds it — that fact lives at the service, and by the time a reply arrives it is already about the past. It cannot even know that its own process has been running continuously since the grant: a paused process observes no gap. So every action taken "while holding the lock" is really taken on a belief about a moment that has passed.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two reasons to take a lock, and only one of them is safe
Martin Kleppmann’s framing is the one to internalise, because it decides everything downstream. You take a distributed lock for one of two reasons.
For efficiency: to stop two workers doing the same expensive job — regenerating a report, warming a cache, running a nightly export. If the lock occasionally fails and two workers run, you waste money and nothing is corrupted. Here an ordinary lock is perfectly adequate, and its failure modes are acceptable by construction.
For correctness: to stop two workers doing something that must happen once — writing a file region, transferring money, allocating an identifier. Here an ordinary lock is not adequate, because the failure mode is corruption and the lock cannot prevent it. You need the resource itself to reject the stale actor, which means Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
The practical instruction is short: decide which one you are doing before you choose a mechanism, and write it in the code comment. Almost every serious distributed-lock incident is an efficiency-grade mechanism deployed against a correctness-grade requirement.
| Purpose | If the lock fails and two run | Mechanism required |
|---|---|---|
| Efficiency — avoid duplicate worktypical | Wasted compute; identical result | Any lock service; failures are acceptable |
| Correctness — avoid duplicate effectprotocol | Corruption, double spend, lost data | Lock **plus** fencing at the resource |
Why a distributed lock is not a mutex
A local mutex is backed by a hardware primitive and a scheduler. Acquisition is atomic; the holder cannot vanish without the runtime knowing; and the mutex is released deterministically when the holder finishes or the process dies. Concurrency owns this material, and spells the distinction out from the single-machine side.
A distributed lock has none of those properties. Acquisition is a network round trip whose result may be lost. The holder can vanish, and the service cannot tell whether it has — so the lock must expire on a timeout, which is a guess. And the holder can be alive but unable to renew, or paused and unaware of the passage of time.
So the lock service is not maintaining mutual exclusion. It is maintaining a *record* of who most recently acquired, with a lease that expires when nobody renews. Mutual exclusion in the world is an inference from that record, and the inference is exactly what fails under a pause.
What a lock service must do to be trustworthy
Not every store that can hold a key can be a lock service. The requirements are specific, and the common shortcuts each break one of them.
The last requirement is the one most often skipped, and skipping it is what turns a correctness lock back into an efficiency lock without anyone noticing.
- Atomic acquisition — a single compare-and-swap-style operation, not read-then-write. Two clients must never both succeed.
- Fault-tolerant and consistent — the lock record must survive node failure without ever being served differently to two clients, which means it must sit behind consensus. A single-node store is a single point of failure *and* a single point of divergence on failover.
- Automatic expiry — because holders die silently, a lock with no lease is a lock that eventually deadlocks the system permanently.
- Owner-checked release — releasing must verify the releaser still holds it, or a slow client will release a lock now held by someone else.
- A monotonic token issued on grant — without which the lock cannot be used for correctness at all.
1# WRONG: release does not check ownership2acquire(key, ttl)3 ... work takes longer than ttl; lock expires; B acquires ...4release(key) # deletes B's lock. B now runs unprotected.5 6# WRONG: read-then-write acquisition is not atomic7if get(key) is None: # two clients can both see None here8 set(key, me, ttl) # and both succeed9 10# RIGHT: atomic acquire, owner-checked release, token returned11token = acquire_atomic(key, owner=me, ttl=30s) # fails if held by another12...13release_if_owner(key, owner=me) # no-op if we no longer hold it14# and every effect carries the token, checked at the resourceThe alternatives that are usually better
Before reaching for a lock service, note that the most common uses of distributed locks have simpler answers that do not couple availability to a second system.
Each of these is worth trying first, because each removes the coupling described in Coordination Couples Availability rather than paying for it.
- A database row lock or unique constraint. If the protected state already lives in one database, that database is already a coordination point you operate well.
SELECT ... FOR UPDATEand a unique index are locks with real transactional semantics. - A single-consumer queue. Exclusivity by construction: one message, one handler, no agreement needed. See Work Queues: One Task, One Worker, Competing Consumers.
- Partitioned ownership. Route all work for a key to one node, so no lock is required at all. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- Idempotence. If running twice is harmless, the lock was protecting nothing that mattered. This dissolves most efficiency-grade uses outright.
- Optimistic concurrency. Compare-and-swap on a version, letting the loser retry. No lock, no expiry, no stale holder — see Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely for why the check must live at the resource.
Key points
- Decide whether the lock is for efficiency or for correctness before choosing a mechanism.
- An efficiency lock may fail occasionally; a correctness lock must be paired with fencing at the resource.
- A distributed lock maintains a record of who acquired most recently, not mutual exclusion in the world.
- Every distributed lock needs an expiry, because the service cannot tell a dead holder from a slow one.
- Acquisition must be atomic and release must be owner-checked; both are commonly got wrong.
- A database constraint, a single-consumer queue or partitioned ownership is usually a better answer.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A client asks the lock service to acquire a named lock with a TTL, atomically.
- • The service grants it if unheld, records the owner and expiry, and returns a monotonically increasing token.
- • The client performs its work, attaching the token to every effect on the protected resource.
- • The client renews the lease periodically if the work outlasts the TTL.
- • On completion the client releases, and the service verifies it is still the recorded owner before deleting.
- • If renewal fails or the lease expires, the service grants to another client with a strictly higher token.
- • The acquire response is lost, so the client believes it failed while the service records it as the holder — the lock is held by nobody until it expires.
- • The client pauses past its lease and continues acting afterwards. See The Stale Lock Holder: A Paused Process Does Not Know It Was Paused.
- • The release request is delayed and arrives after another client has acquired, releasing someone else’s lock.
- • The lock service is unreachable, so no work proceeds at all — the availability coupling made concrete.
- • The lock service fails over and, if not consensus-backed, serves inconsistent state to two clients.
- • Renewal succeeds at the service but the reply is lost, so the client abandons work it was entitled to continue.
- • Two workers, no errors: a pause or a lost renewal produces two holders and neither logs anything unusual. The operator sees duplicated output, doubled side effects, or corrupt data, with both workers reporting clean runs.
- • Lock held by a ghost: a client crashed after acquiring; nothing runs until the TTL expires. The operator sees a job that has not run for exactly the lease duration and no process holding anything.
- • Released someone else’s lock: a slow client releases after expiry, and a third client immediately acquires while the second is mid-work. The operator sees interleaved work from two workers and a lock that changed hands without an expiry event.
- • Total stall on lock-service outage: the lock service is unreachable and every guarded operation fails. The operator sees healthy workers, healthy target systems, and an idle pipeline.
- • Lease thrash under load: work regularly outlasts the TTL and renewals compete with the work for resources, so locks expire mid-job repeatedly. The operator sees rising expiry rate correlated with load, and increasing duplicate execution.
- • Every acquisition is a coordination point: the client cannot proceed without the service, so its availability is now coupled to it.
- • Renewals extend that coupling for the whole duration of the work, not just its start.
- • Fencing moves the safety check to the resource, so a lost renewal produces a rejected write rather than a corrupted one — coordination for the grant, local checking for the act.
- • A consensus-backed lock service preserves at-most-one-holder-of-record through node failures, and becomes unavailable rather than inconsistent when it loses quorum.
- • A non-consensus lock service can serve two holders after a failover, which is the most common way a "correctness" lock silently becomes an efficiency lock.
- • Without fencing, at-most-one-holder-of-record never implies at-most-one-actor under any failure.
- • Detect: monitor lease expiries that occur while work is still running — this is the direct precursor to double execution.
- • Contain: fence every effect; make the work idempotent where you can; keep lease TTLs comfortably above the p99 work duration.
- • Recover: on lost renewal the holder must stop immediately rather than finish "just this bit", and re-acquire before resuming.
- • Reconcile: audit for duplicate effects across every expiry event, since that is the window where they occur.
- • Verify: test the pause case deliberately — suspend a holder past its lease and confirm the resource rejects its later writes.
- • Lock acquisition failure rate and wait time, per lock name.
- • Lease expiries occurring while the holder was still working — the double-execution precursor.
- • Distribution of work duration against lease TTL; the overlap is your exposure.
- • Renewal failure rate and what the holder did afterwards.
- • Rejected-token counts at the protected resource, which is the only direct evidence fencing is working.
- • Preventing duplicate expensive work across a fleet, where an occasional overlap is merely wasteful.
- • Coordinating a singleton process — one scheduler, one compactor, one migration runner — with fencing where the effect matters.
- • When the protected state spans systems and no single database can hold the constraint.
- • When taken for correctness without fencing — the most damaging misuse in this module.
- • On a high-frequency path, where every operation now depends on a second system being reachable.
- • When the protected state already lives in one database that could enforce the constraint transactionally.
- • For long-running work, where lease renewal becomes its own source of failure.
- • A database transaction, row lock or unique constraint, when the state lives in one database. Simpler, transactional, and already operated.
- • A single-consumer queue, giving exclusivity structurally with no lock at all. See Work Queues: One Task, One Worker, Competing Consumers.
- • Partitioned ownership so each key has one owner and no lock is needed. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- • Optimistic concurrency: act, then commit conditionally on an unchanged version, and retry on conflict.
- • Idempotent operations, which remove the need for exclusion entirely for efficiency-grade cases.
Efficiency lock or correctness lock?
What people believe, and what is true
A distributed lock gives mutual exclusion like a mutex.
It gives at most one *record* of ownership. Mutual exclusion among actors is an inference, and the inference fails whenever a holder is paused or partitioned.
A single Redis instance is fine for locking.
It is fine for efficiency locks. On failover it can hand the same lock to two clients, so it provides no correctness guarantee — a distinction its own documentation is explicit about.
A longer TTL makes the lock safer.
It makes expiry-during-work rarer and every crash-induced stall longer. The unsafe window never closes; only fencing removes it.
If I check the lock is still mine before writing, I am safe.
The pause can occur between the check and the write. Verify-then-act is not atomic across a network — the check must live at the resource.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A distributed lock records who most recently acquired, with an expiry. Use it freely to avoid duplicate work. Use it for correctness only when the protected resource also checks a fencing token.
Practical
Choose an atomic acquire, an owner-checked release, a TTL above p99 work duration, and a token on every effect. Monitor expiries that happen while work is running. Consider a database constraint or a single-consumer queue first — either removes the availability coupling entirely.
Advanced
The lock service can only ever provide a consistent *record*, because the gap between the record and the world contains the client’s process, its scheduler and the network. Every attempt to close that gap by timing — longer leases, pre-write checks, tighter clocks — narrows the window without eliminating it. Only relocating the check to the point where the effect lands changes the class of guarantee, from probabilistic to absolute.
Apply it
- 🔧 Audit a distributed lock in your system: classify it as efficiency or correctness, then check whether the mechanism matches the classification.
- 🔧 Redesign a locked operation so it needs no lock, using ownership, a queue, or idempotence.
- 💬 What is the difference between using a distributed lock for efficiency and for correctness?
- 💬 Why is a distributed lock not equivalent to a mutex?
- 💬 What must a lock service provide for you to rely on it, and which of those does a single Redis node lack?
- 💬 Name two bugs that appear in almost every hand-rolled lock implementation.