The question this answers
A process holding a lock stalls for a minute and then wakes up. What does it believe, and what does it do?
Without fencing: none. The lock service correctly records a single holder while two processes act on the resource. With fencing at the resource: at most one actor can successfully affect the resource, regardless of pause duration, clock behaviour or partition — the pause becomes a rejected write rather than a corruption.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
This lesson is the sharpest case in the domain. A paused process observes nothing at all — no gap, no signal, no elapsed time. On resumption its next instruction executes as though the previous one had just completed. It does not know its lease expired, that another holder exists, or that any time passed, unless it explicitly re-reads a monotonic clock and even then only if it thinks to. Every local variable, including "I hold the lock", is still there and still says what it said.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The sequence, step by step
Worker A acquires a 30-second lease on shard-7 and begins writing. Two seconds in, the runtime begins a stop-the-world collection — or the hypervisor suspends the VM, or the container is throttled to near-zero CPU. A stops executing. It does not log anything, because logging requires executing.
Forty-five seconds later the lock service, which has received no renewal, expires A’s lease. Worker B acquires shard-7 and begins writing. B is entirely correct: the lock service told it the lock was free, and it was.
A resumes. Its program counter is exactly where it was; its stack says it holds the lock; its next statement is the write it was about to perform. It performs it. Two workers are now writing to shard-7, both believing they hold exclusive access, and the lock service’s records are perfectly accurate throughout.
The crucial observation is that no component behaved incorrectly. A followed its code, B followed its code, the lock service followed its protocol. The defect is architectural: the design assumed that holding a lock at time T implies holding it at time T + ε, and across a machine boundary that implication does not hold.
Why the obvious fixes do not work
Every team meets this problem and proposes the same four remedies. Each of them narrows the window and none of them closes it, which is exactly the distinction that matters when the consequence is data corruption.
The pattern across all four is identical: they are attempts to make the holder’s belief reliable. The belief cannot be made reliable, because the holder is the one component that is guaranteed to be uninformed during precisely the interval in question.
- "Make the lease longer." Now it takes a longer pause. Multi-minute GC pauses and VM suspensions both exist, and every crash now stalls the system for the full lease. You have traded a rare corruption for a routine outage and kept the corruption.
- "Check the lock before writing." The pause can occur between the check and the write. Verify-then-act is not atomic across a network — this is the same impossibility as A Timeout Tells You Nothing About Whether It Happened, and no amount of re-checking removes it.
- "Tune the GC / add more memory." Reduces frequency, not possibility. And it addresses only one cause: VM suspension, CPU throttling, disk stalls and network partitions produce identical behaviour.
- "Detect the pause and abort." A useful defence — compare a monotonic clock before each effect — but it runs *after* the pause, and the effect may already be in flight. It reduces exposure; it does not create a guarantee.
The resolution: move the check to where the effect lands
Since the holder cannot know whether it still holds authority, stop asking it to. The resource knows which token it last accepted, and the resource is where the effect actually takes place. Give every grant a monotonically increasing token, attach it to every write, and have the resource reject anything below its high-water mark, atomically with the write.
Now A’s resumption is harmless. Its write carries token 33, the resource has seen 34, and the write is refused. A learns it has been superseded — from the rejection, which is the only channel that could have told it. This is what "safe rather than merely unlikely" means: the correctness argument no longer contains a sentence about how long a pause might last.
The full mechanism, its granularity choices and the resources that cannot support it are in Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely. What belongs here is the reasoning that forces it: the holder is structurally incapable of knowing, so the check must live elsewhere, and the only "elsewhere" that is guaranteed to be involved in every effect is the resource itself.
| Approach | Closes the window? | What it costs |
|---|---|---|
| Longer leaseprotocol | No — needs a longer pause | Longer stall on every genuine crash |
| Re-check before writeprotocol | No — pause can land between | An extra round trip per write |
| GC tuning / bigger heaptypical | No — reduces frequency only | Addresses one cause of several |
| Monotonic pause detectiontypical | No — acts after the fact | Cheap; a good defence in depth |
| Fencing token at the resourceprotocol | **Yes** — timing-independent | The resource must check; some cannot |
| Idempotent effecttypical | Not applicable — makes it harmless | Only possible for some effects |
How it looks in production, and why nobody notices
This class of incident is unusually hard to attribute, and the reason is worth stating: every participant logs a successful, ordinary run. A logs "acquired lock, did work, finished". B logs the same. The lock service logs a normal expiry and a normal grant. There is no exception, no timeout, no error rate to alert on.
What you see instead is downstream and delayed: a file with interleaved content, a counter that is wrong by a small amount, two invoices, a segment overwritten. The gap between the event and the symptom is often days, by which time the logs have rotated.
The signals that actually catch it are proactive: lease expiries that occur while the holder is still working (the precursor), pause durations measured by the holder itself with a monotonic clock, and rejected-token counts at the resource (evidence the fencing is doing its job). None of these are default metrics; all three are cheap.
worker-a 10:14:02 acquired lock shard-7 (lease 30s, token 33) worker-a 10:14:04 writing segment 0004 lockd 10:14:34 lease shard-7 expired (no renewal since 10:14:02) worker-b 10:14:35 acquired lock shard-7 (lease 30s, token 34) worker-b 10:14:37 writing segment 0004 worker-a 10:14:47 writing segment 0004 <-- resumed after a 43s pause worker-a 10:14:48 released lock shard-7 <-- released B's lock worker-a 10:14:48 job complete worker-b 10:15:05 job complete # both jobs "succeeded"; segment 0004 contains bytes from both
Key points
- A paused process observes no elapsed time and resumes with every belief intact, including "I hold the lock".
- The lock service can be perfectly correct while two processes act on the resource.
- Longer leases, pre-write checks and GC tuning narrow the window; none of them close it.
- The resolution is to move the authority check to the resource, via a fencing token.
- The correctness argument must not contain a sentence about how long a pause might last.
- The incident produces no errors — detect it with expiry-during-work, self-measured pause duration, and rejected-token counts.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A acquires the lock and receives a lease and a token.
- • A is descheduled — GC, hypervisor suspension, CPU throttling, or a network partition from the issuer.
- • The lease expires at the issuer because no renewal arrives; no detection of A is required.
- • B acquires the lock with a strictly higher token and begins acting.
- • A resumes with unchanged local state and continues from its next instruction.
- • Both act. Whether that is harmful depends entirely on whether the resource checks tokens.
- • A stop-the-world garbage collection lasting longer than the lease.
- • A virtual machine suspended and later resumed, or live-migrated with a stall.
- • A container throttled to near-zero CPU by a cgroup quota.
- • A disk stall or swap storm making the process unschedulable in practice.
- • A network partition between holder and issuer, producing the same outcome with no pause at all.
- • A slow release arriving after expiry, deleting the new holder’s lock.
- • Interleaved writes with two clean logs: the operator sees a corrupted file or record and two workers each reporting a successful run, minutes apart, with an expiry between them.
- • Released someone else’s lock: the resumed holder calls release and removes the current holder’s lock, so a third worker starts while the second is mid-flight. The operator sees a lock changing hands with no expiry event.
- • Duplicate external effects: two payouts, two exports, two emails. The operator sees duplicates with distinct request ids and no errors — usually reported by a customer rather than by monitoring.
- • Silent counter drift: both holders increment a shared counter, producing a value that is wrong by a small amount and never reconciles. The operator sees a slow divergence between two systems’ totals with no event to attach it to.
- • Rejection storm after fencing is added: the resumed holder retries its stale-token write in a loop. The operator sees a burst of rejections from one client — which is the mechanism working, and the holder failing to stop on rejection.
- • The lock provided coordination at acquire time; the failure is that the design assumed that coordination persisted through the pause.
- • Fencing does not add coordination — it adds a local check at the resource, which costs nothing and works during partitions.
- • This is the clearest instance of the module’s theme: coordinate once to obtain authority, verify locally where the effect lands.
- • With fencing, the resource remains correct through any pause of any length; the stale actor simply fails.
- • Without fencing, the resource is corrupted and the lock service is no help — it was never wrong.
- • Effects on unfenceable systems remain exposed regardless, which is why idempotence is the fallback.
- • Detect: alert on lease expiry while the holder still reports work in progress, and on self-measured pauses exceeding a fraction of the lease.
- • Contain: fence every effect; make the holder check a monotonic clock before each effect and abort if the deadline passed; use owner-checked release.
- • Recover: the stale holder must stop on the first rejection, re-acquire, and obtain a higher token before resuming.
- • Reconcile: audit for duplicate or interleaved effects across every expiry boundary — that is the entire search space.
- • Verify: deliberately pause a holder past its lease in a test environment and confirm the resource rejects its subsequent writes.
- • Lease expiries that occur while the holder believes it is working — the leading indicator.
- • Holder-measured pause durations from a monotonic clock, as a distribution.
- • Rejected-token counts at the resource, by client — proof the fencing path is exercised.
- • Runtime pause metrics (GC pause time, CPU throttling, steal time) alongside lease TTL on the same graph.
- • Duplicate-effect counters keyed by job or lock name.
- • Understanding this is what makes the case for fencing tokens concrete rather than theoretical.
- • It is the argument that ends the "just increase the timeout" conversation, permanently.
- • It generalises to every timeout-based authority: sessions, leader leases, ownership assignment.
- • When it produces paralysis — for efficiency-grade locks, an occasional double run is exactly the acceptable outcome the lock was chosen for.
- • When teams add fencing to the primary write path only, leaving caches, webhooks and exports unprotected while believing the problem is solved.
- • Make the effect idempotent so a second application converges to the same state — the only option when the resource cannot check a token.
- • Route work through a single-consumer queue so exclusivity is structural and there is no lease to expire. See Work Queues: One Task, One Worker, Competing Consumers.
- • Partition ownership so there is only ever one candidate actor per key. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- • Have the holder abort on detecting a pause via a monotonic clock — a real reduction in exposure, and not a guarantee.
A paused process does not know it was paused
holdsLock = true // never changed; nothing changed it leaseToken = 41 // the only token A has ever had acquiredAtWall = 12:04:01 // still says what it said // A's next instruction executes as though the previous one // had just completed. There is no gap to observe, no signal, // and no elapsed time — unless A explicitly re-reads a // monotonic clock, and even then only if it thinks to.
What people believe, and what is true
This is rare enough to ignore.
It is rare per operation and certain at scale. A one-in-a-million window at ten thousand operations per second occurs several times a day.
The lock service had a bug.
It behaved correctly throughout. The defect is the assumption that a grant remains true while the holder is not executing.
The process will notice it was paused.
Only if it explicitly re-reads a monotonic clock and checks. Nothing in the runtime tells it, and its own state is unchanged.
It only happens with garbage collection.
VM suspension, CPU throttling, disk stalls and network partitions all produce identical behaviour. Non-GC languages are not exempt.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A process holding a lock can pause long enough for its lease to expire, then resume with no idea anything happened. Two processes then act. The fix is to have the resource reject the old holder, not to make the lease longer.
Practical
Attach a token to every effect and check it at the resource. Monitor expiries that occur during work, and have the holder measure its own pauses with a monotonic clock and abort. Use owner-checked release. Then test it by actually suspending a holder past its lease.
Advanced
The general principle is that authority granted at time T cannot be assumed to hold at T + ε when ε is unbounded and unobservable to the holder. Any correctness argument containing "the pause will be shorter than the lease" is an assumption about the world rather than a property of the system. Relocating the check to the point of effect replaces that assumption with an integer comparison, which is the only move in this space that changes the class of guarantee rather than its probability.
Apply it
- 🔧 Reproduce the scenario deliberately: pause a holder past its lease and observe the resource with and without a token check.
- 🔧 List every effect a locked job in your system performs and mark which are fenced, which are idempotent, and which are exposed.
- ⚡ A compaction job holds a lock and writes segments to object storage. It is throttled for 90 seconds. Describe what the storage bucket contains afterwards, with and without conditional writes.
- 💬 Walk me through what happens when a lock holder pauses for longer than its lease.
- 💬 Why does increasing the lease duration not fix it?
- 💬 Why can the client not just check that it still holds the lock before writing?
- 💬 What would you monitor to know this is happening in your system today?