The question this answers
How do I stop a node that lost its lock — but does not know it — from corrupting the resource it was protecting?
Given a monotonically increasing token issued by a single authority, and a resource that persists the highest token it has accepted and rejects any write carrying a lower one, at most one writer can ever succeed at a time, regardless of clocks, pauses, or partitions. Safety here is independent of timing entirely — which is what distinguishes it from every lease-tuning approach.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A holder knows the token it was given. It does not know whether that token is still the highest one issued — that fact lives at the issuer and at the resource, not at the holder. The crucial design move is that the holder is never asked to know: it simply presents its token, and the resource, which does know, decides. Authority is verified at the place where the effect lands.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The window you cannot close
A lock service grants A a lease for 30 seconds. A pauses for 45 — garbage collection, a suspended VM, a saturated disk. The lease expires; the service grants the lock to B; B starts writing. A resumes, entirely unaware that any time has passed, and writes too. Two writers, and neither has done anything wrong.
The instinct is to make the lease longer, or to have A check the clock before writing. Neither works. A longer lease widens the outage when a holder genuinely dies, and it does not eliminate the case — it only requires a longer pause. And a clock check before writing is useless because the pause can occur between the check and the write; there is no way to make "verify then act" atomic across a network.
So the window is structural. The only remaining move is to stop trying to prevent A from writing, and instead make A’s write fail at the destination.
The mechanism is three lines
The issuer maintains a counter that increases on every grant. The holder passes the token with every operation. The resource keeps the highest token it has seen and rejects anything below it, atomically with the write itself.
The atomicity is not optional. If the resource checks the token, then performs the write as a separate step, a stale write can slip between the two — you have reproduced the original problem at a smaller scale. The check and the write must be one operation: a conditional update, a compare-and-swap, an UPDATE ... WHERE token >= :token, or a storage system with native preconditions.
This is Terms and Epochs: Making Stale Leaders Harmless generalised. A Raft term protects the log because log participants check terms; a fencing token protects arbitrary resources because you have taught *them* to check. Same integer, same comparison, different place.
1# Issuer (lock service, or a consensus group)2grant(lock_name, holder):3 token = atomic_increment(counter[lock_name]) # strictly monotonic4 return token5 6# Holder7token = lock_service.acquire("shard-7")8... arbitrary delay: GC, VM suspend, network stall ...9storage.write("shard-7", data, token) # holder makes no timing claim10 11# Resource — this is the part everyone forgets to build12write(key, data, token):13 ATOMIC:14 if token < highest_seen[key]:15 return REJECT(highest_seen[key]) # stale writer, bounced16 highest_seen[key] = token17 persist(key, data)18 return OKWhy "the resource must participate" is the hard part
The mechanism is trivial; the deployment is not, because it requires cooperation from the thing you are writing to. If your resource is a database row, this is easy — a conditional update does it. If it is your own service, you add a column. If it is a POSIX filesystem on a shared volume, there is nowhere to put the check, and fencing must move down to the storage layer (SCSI reservations, or the network fabric cutting the node off entirely — the older meaning of "fencing", as in STONITH).
If the resource is a third-party API with no conditional write, you cannot fence it. That is a real constraint and it should change the design: either make the effect idempotent so a duplicate is harmless, or accept that this operation cannot be made exactly-once and reconcile afterwards. What you must not do is pretend a longer lease solved it.
A useful test when reviewing a design: name the line of code at the resource that compares two integers. If nobody can point at it, the system has a lock but no fencing, and the lock is a performance optimisation rather than a safety mechanism.
| Resource | Where the check lives | Feasible? |
|---|---|---|
| Row in your databaseprotocol | `UPDATE ... WHERE fence_token < :t` | Yes — trivial |
| Your own serviceprotocol | Highest-token column, checked in the handler | Yes |
| Object store with preconditionstypical | Conditional PUT on an ETag or version | Yes, if the API offers it |
| Shared block device / POSIX FStypical | No application-level place to check | Only via storage-level reservation or STONITH |
| Third-party API without conditionalstypical | Nowhere | No — make the effect idempotent instead |
Token order must come from one authority
The tokens must be strictly increasing and issued by a single authority, which is the sense in which fencing depends on consensus rather than replacing it. Two independent issuers can hand out the same number, or hand out numbers whose order does not reflect the order of grants, and the resource’s comparison becomes meaningless.
This is why a timestamp is a bad token: two machines can produce the same or out-of-order values under Clock Skew: The Gap You Cannot Measure From Inside, and a clock that steps backwards produces a token that will be rejected forever, locking you out of your own resource. A Raft term, a ZooKeeper zxid, an etcd revision, or a database sequence all work because a single agreement process orders them.
One consequence worth knowing: because the resource stores the highest token it has seen, a monotonic counter that resets — a redeployed lock service with a fresh in-memory counter — makes every subsequent write fail. The counter must be as durable as the resource’s memory of it.
Key points
- The window where a stale holder still believes it holds the lock cannot be closed by tuning.
- A fencing token makes the window harmless: the resource rejects writes carrying a lower token.
- The check and the write must be atomic at the resource, or you have reintroduced the race.
- Safety becomes independent of clocks, pauses and partitions — this is what "safe, not merely unlikely" means.
- Tokens must be strictly increasing and issued by one authority; timestamps do not qualify.
- If the resource cannot check a token, the operation cannot be fenced — make it idempotent instead.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • An issuer with a single, ordered source of truth increments a counter on every grant.
- • The grant returns the token to the holder along with the lock or lease.
- • The holder attaches the token to every operation it performs on the protected resource.
- • The resource compares the presented token against the highest it has persisted for that key.
- • Lower or equal-but-stale tokens are rejected; higher tokens are accepted and become the new high-water mark, atomically with the write.
- • The rejected holder learns it has been superseded from the rejection itself, and stops.
- • The resource does not implement the check, so the token is decorative.
- • The check and the write are separate operations, leaving a race between them.
- • The issuer’s counter is not durable and resets, so new tokens are below the resource’s high-water mark.
- • Two issuers exist — a redeployed lock service, or two lock services — producing incomparable tokens.
- • The high-water mark is stored per-service rather than per-key, so unrelated keys interfere.
- • The holder retries a rejected write with the same token, turning a clean rejection into a hot loop.
- • Fenced-out forever: the lock service was redeployed with a reset counter and every write is now rejected. The operator sees 100% write rejection with a healthy lock service and healthy clients, and log lines showing tokens far below the stored maximum.
- • Silent double-write: fencing was specified but the resource never implemented the comparison. The operator sees interleaved writes from two nodes and a corrupted result, with both nodes reporting success — the failure looks exactly like having no lock at all, because it is.
- • Rejection storm after a pause: a resumed holder retries its stale-token write in a tight loop. The operator sees a spike of
REJECTresponses from one client id and elevated load on the resource. - • Partial fencing: the database write is fenced but the cache invalidation and the outbound webhook are not. The operator sees consistent primary data with a stale cache and duplicate webhooks — the classic incomplete adoption.
- • Cross-key interference: a single global high-water mark causes writes to key X to be rejected because key Y advanced. The operator sees rejections that correlate with unrelated traffic.
- • Coordination happens once, at grant time, and produces the token. Nothing coordinates at write time.
- • That relocation is the whole value: the expensive agreement is amortised over many operations, and the cheap integer comparison happens on the hot path.
- • The resource becomes a second point of serialization per key — which is fine, because it was already the point where writes are ordered.
- • A stale holder can attempt anything and succeed at nothing on a fenced resource.
- • A partition between holder and issuer does not compromise safety; the holder simply cannot renew and its token ages out naturally.
- • Effects on unfenced resources are unaffected and remain the system’s exposure.
- • Detect: count rejected-token responses per resource and per client — a non-zero rate is the system working, a sustained one is an incident.
- • Contain: on rejection, the holder must stop and re-acquire, never retry with the same token.
- • Recover: re-acquire the lock, obtain a fresh (higher) token, resume.
- • Reconcile: for effects that landed on unfenced resources during the window, reconcile explicitly — see Reconciliation Is a Component, Not a Cleanup Script.
- • Verify: audit that every write path to the protected resource carries and checks a token, including admin tools and migration scripts, which are the usual bypass.
- • Rejected-write count by token, resource and client identity.
- • Current high-water token per key versus the issuer’s counter — divergence signals a reset or a second issuer.
- • Distribution of the gap between issued and presented tokens, which measures how stale holders actually get.
- • Whether the issuer’s counter is on durable storage — an audit, and the difference between an outage and a total lockout.
- • Whenever a lock or lease protects an effect on a resource that can check a token: a shard owner writing to storage, a single-writer job, a leader compacting files.
- • Whenever pauses are plausible — any managed runtime with stop-the-world GC, any virtualised or containerised workload.
- • Whenever the cost of a double write is higher than the cost of a rejected write, which is nearly always.
- • When the protected effect is naturally idempotent, in which case the token is machinery protecting against a harmless duplicate.
- • When it is applied to only some of the write paths, giving the confidence of fencing with the exposure of none.
- • When the resource cannot support an atomic conditional write and the check is bolted on as a separate read — worse than nothing, because it looks correct.
- • Make the operation idempotent so a duplicate write converges to the same state — no token needed, and it is the only option for unfenceable third-party effects.
- • Compare-and-swap on the data’s own version rather than on a lock generation: protects each write individually without a lock concept. API Design owns the endpoint-level form of this.
- • Storage-level or fabric-level fencing (SCSI reservations, STONITH, cutting the node’s network) when the resource has no application layer to check anything.
- • Single-writer-by-partitioning: route all writes for a key through one owner so there is never a second writer to fence. Cheaper, and only as strong as the routing layer.
Fencing: making the stale actor safe
value = segment-2 (by B, token 42) high-water = 42 last write = A
What people believe, and what is true
A longer lease removes the need for fencing.
It requires a longer pause to trigger the bug, and lengthens every genuine failover. The window never reaches zero.
The client can check whether its lease is still valid before writing.
The pause can happen between the check and the write. Verify-then-act is not atomic across a network — which is why the check must live at the resource.
A timestamp works as a fencing token.
Two machines can emit equal or out-of-order timestamps, and a backwards clock step produces tokens that are rejected forever. Tokens need a single ordering authority.
We use a distributed lock, so we are fenced.
A lock tells the holder it may proceed. Fencing tells the resource whom to believe. Without the resource-side check you have coordination without protection.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Give every lock grant an increasing number. Send it with every write. Have the resource remember the highest number it has accepted and reject anything lower. A stale holder’s writes then fail harmlessly.
Practical
Pick the token source (Raft term, etcd revision, DB sequence), thread it through every write path including admin tools, and implement the check as a single conditional update. On rejection, stop and re-acquire — never retry the same token. Monitor rejection counts and make sure the issuer’s counter is durable.
Advanced
Fencing converts a timing-dependent safety argument into a timing-independent one. Before it, correctness rests on "the pause is shorter than the lease", which is an assumption about the world you cannot verify. After it, correctness rests on integer comparison and atomic update, which are properties of your code. That is the same move Terms and Epochs: Making Stale Leaders Harmless makes inside the cluster, and it is the reason both mechanisms keep working in exactly the conditions where detection-based approaches fail.
Internals
Granularity is the real design decision. A per-resource high-water mark serialises unrelated work; a per-key mark multiplies the state the resource must persist and complicates operations spanning keys. Systems commonly use the lock’s scope as the granularity — one mark per shard, per partition, per document — and then accept that a multi-key operation needs either one enclosing lock or per-key tokens checked in a single transaction. Where the store offers native preconditions (conditional PUT on version, IF clauses), prefer them: they make the atomicity the store’s problem rather than yours.
Apply it
- 🔧 Take a system you know that uses a distributed lock and identify the exact line that would compare tokens. If there is none, describe what a stale holder could do.
- 🔧 Design fencing for an operation that spans two resources, and state what you can and cannot guarantee.
- ⚡ A shard owner writes compacted segments to object storage. Ownership moves during a GC pause. Show how fencing prevents the old owner from overwriting the new owner’s segment, and what happens if the object store offers no conditional put.
- 💬 A client holding a distributed lock pauses for a minute. How do you keep it from corrupting the resource when it wakes up?
- 💬 Why is a longer lease not a fix?
- 💬 What must the resource do for a fencing token to mean anything?
- 💬 Why is a wall-clock timestamp a poor fencing token?