The question this answers
How do I tell whether that node has crashed or is merely slow?
You cannot. In an asynchronous system there is no algorithm that distinguishes a crashed process from a slow one, because the only evidence available — the absence of a message — is produced identically by both. What a system can guarantee is that acting on the wrong answer is safe, and that is a different and achievable goal.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The observer knows the elapsed time since the last message. The *paused* node knows nothing at all — it is not aware that time passed, so on resumption it continues from exactly where it was, with every belief it held intact, including beliefs about locks it holds and roles it occupies. That asymmetry is the whole danger.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Why it is undecidable, plainly
Suppose you had an algorithm that, given the absence of messages from a node, correctly reported whether it had crashed. Now consider a node that has not crashed but whose next message will be delayed by exactly one second longer than your algorithm waits. The evidence available to the algorithm is identical in both cases: nothing arrived. So the algorithm returns the same answer in both cases, and in one of them it is wrong. Since the asynchronous model places no bound on delay, this construction is always available. There is no algorithm. It is not a matter of better instrumentation or a smarter heuristic.
This is worth stating as bluntly as possible because the intuition from single-machine debugging fights it. Locally, a stopped process is observable: the OS knows, ps knows, the exit code exists. Remotely there is no oracle — the OS that knows is on the other side of the thing that is not working. The only channel through which the fact could reach you is the channel whose failure you are trying to diagnose.
What real systems have is partial synchrony: delays are usually bounded, so a timeout is usually right. "Usually" is doing an enormous amount of work in that sentence, and the moments when it fails are precisely the moments of high load, which are precisely the moments when a wrong failover is most damaging.
Pauses are longer and more common than people expect
Engineers underestimate this because the pauses that matter are invisible from inside the process. A stop-the-world garbage collection on a large heap can exceed several seconds. A virtual machine can be live-migrated or suspended, resuming minutes later with no notification to the guest. Memory pressure can push a process into swap, turning microsecond operations into millisecond ones. CPU quota throttling in a container can stop a process for a scheduling period at a time, repeatedly. A synchronous disk write to a degraded volume can block for tens of seconds. Even a laptop lid closing does this, which is why the behaviour shows up in local development as "it worked, then everything was weird".
In every one of these the process is unaware. There is no callback for "you were suspended". Wall-clock time can be consulted after the fact, which is the basis of one useful mitigation — check whether more time has elapsed than expected before performing a privileged action — but nothing prevents the pause from occurring *between* that check and the action.
This is why the honest design position is not "make pauses shorter", useful though that is. It is assume every process may pause at any moment for an arbitrary duration, and make that safe.
- Stop-the-world GC: seconds on a large heap; longer if the heap is under pressure.
- VM live migration or suspend/resume: no guest notification, arbitrary duration.
- Swap and memory pressure: everything slows by orders of magnitude at once.
- CPU quota throttling: repeated stalls of up to a scheduling period, invisible in CPU-percent metrics.
- Blocking I/O on a degraded volume: tens of seconds, in code that looks synchronous and cheap.
The design response: make being wrong safe
Since detection cannot be made correct, correctness has to come from somewhere else. The standard construction is a lease plus a fencing token. A worker holds a lease that expires; to act on a shared resource it presents a token that increases every time the lease is granted; the resource itself refuses any operation carrying a token lower than the highest it has seen. Now a paused worker that resumes and writes is rejected *by the storage layer*, without anyone having to know whether it was crashed or slow.
The important property of that design is where the check lives. If the worker checks its own lease before writing, the pause can happen between the check and the write and the design provides nothing. If the *resource* checks the token as part of the write, there is no window — the check and the effect are atomic at the place that matters. This is the general shape of every robust answer in this area: move the check to the point of effect.
The same reasoning is why an expired lease should be treated as expired by its holder *earlier* than by its granter. The holder cannot trust its own clock relative to the granter’s, so it builds in a margin and stops acting before the granter would consider the lease free. That does not make it safe on its own — a pause defeats it — but it narrows the window that fencing has to cover.
1# Unsafe: the pause can land between the check and the write.2if lock.still_held(): # true at this instant3 # ... 30-second GC pause ... lease expired, someone else took over4 storage.write(data) # accepted. Two writers.5 6# Safe: the resource enforces it, atomically with the effect.7token = lock.acquire() # monotonically increasing: 41, 42, 43...8# ... 30-second GC pause ... someone else acquired token 429storage.write(data, fence=token)10# storage has seen 42; rejects 41. No detection required, no window.Distinguishing them after the fact
You cannot decide it in the moment, but you can usually determine it afterwards, and this matters for operations. A crashed node has a process start time later than the incident, an empty in-memory state, and typically an entry in the kernel log or the orchestrator’s event stream. A paused node has continuous uptime spanning the incident, a GC log or throttling metric covering the gap, and — the clearest signal — application logs that resume mid-operation rather than at startup.
The single most useful instrumentation is a pause detector: a thread that sleeps for a fixed short interval in a loop and records when it wakes up later than expected. The gap between expected and actual wake-up is the pause, measured from inside the process, and it catches every cause at once — GC, swap, throttling, hypervisor suspension — without needing to know which one it was. It costs almost nothing and it converts the most confusing class of incident into a number on a graph.
This distinction changes the remedy entirely, which is why it is worth determining. A crash points at the process: a bug, an OOM kill, a bad deploy. A pause points at the environment: heap sizing, memory limits, CPU quota, storage latency, a noisy neighbour. Teams that cannot tell the two apart tend to apply crash remedies to pause problems, which is how you get a service that is restarted repeatedly and keeps stalling.
Key points
- Distinguishing a crashed node from a slow one is undecidable in an asynchronous network — not hard, undecidable.
- A paused process does not experience the pause; it resumes with every belief intact, including beliefs about locks and roles.
- Pauses of seconds to minutes are routine: GC, swap, CPU throttling, VM migration, blocking I/O.
- The answer is not better detection but making a wrong decision safe — leases plus fencing checked at the point of effect.
- A pause detector inside the process turns the most confusing incident class into a measurable number.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • An observer waits for a message and does not receive one within its threshold.
- • The observer must act, and has no evidence distinguishing crash from delay.
- • It acts — evicts, fails over, reassigns — on the assumption of death.
- • The target, if merely paused, resumes with no knowledge that anything happened and continues its previous work.
- • The only thing preventing two actors from proceeding is a check performed at the resource, on a token that ordering guarantees the stale actor cannot have.
- • A pause exceeds every timeout in the system simultaneously, so all observers agree — wrongly.
- • A node resumes and completes a write that logically belongs to a superseded epoch.
- • The lease holder checks its own lease, then pauses, then acts.
- • The fencing token is issued but the resource does not check it, so the mechanism exists on paper only.
- • A restart is applied to a pause problem, clearing the symptom and leaving the cause.
- • Double execution: a paused job resumes after its replacement has started. The operator sees the same job id producing two sets of output, with timestamps separated by roughly the pause duration.
- • Unfenced late write: an evicted node writes after eviction. The operator sees a record modified by an instance that the orchestrator had already terminated, and a version history with an inexplicable gap.
- • Restart loop on a pause: repeated liveness failures cause repeated restarts, each one re-warming caches and making the next pause worse. The operator sees a restart counter climbing and CPU throttling metrics nobody is looking at.
- • Correlated pause: garbage collection or throttling hits many instances at once, so a whole tier is suspected simultaneously. The operator sees a fleet-wide health drop with no deploy and no dependency incident.
- • No coordination can resolve the question — it is a property of the evidence, not of the protocol.
- • Coordination is used instead to make the *consequences* safe: a majority agrees on an epoch, and the epoch number becomes the fence.
- • The cost is that every ownership transfer requires agreement, and the storage layer must participate by enforcing the token.
- • A fenced system remains safe regardless of how wrong the suspicion was — the late writer is simply rejected.
- • An unfenced system remains available and offers no guarantee at all during the window, which may be minutes.
- • The paused node’s local state stays valid from its own perspective throughout, which is why it acts with confidence.
- • Detect: run a pause detector in every process and alert on gaps beyond a threshold.
- • Contain: ensure every exclusive action carries a token that the resource validates.
- • Recover: after a pause, have the process re-validate its authority before resuming work rather than continuing blindly.
- • Reconcile: look for duplicated effects during the window between eviction and the paused node’s return.
- • Verify: test it — pause a process with a signal and confirm its subsequent writes are rejected rather than accepted.
- • In-process pause duration from a wake-up-gap detector, which catches GC, swap, throttling and hypervisor suspension in one signal.
- • CPU throttling counters, which are absent from CPU-utilisation graphs and are a leading cause of mysterious stalls in containers.
- • Rejected fenced operations, broken down by token gap — a large gap means a long pause and tells you the detector threshold is too aggressive or the heap too large.
- • Process uptime at the moment of an incident, which is the fastest way to tell a crash from a pause after the fact.
- • Anywhere a node holds an exclusive role: a leader, a lock, a partition owner, a singleton job.
- • Anywhere a timeout triggers a consequential action, especially automatic failover.
- • For stateless request handling where a duplicate is harmless, reasoning about pauses adds machinery with no risk to mitigate.
- • Fencing everything, including operations that are naturally idempotent, adds a dependency on the token issuer to paths that did not need one.
- • Remove exclusivity: design the operation so concurrent execution by two workers is harmless, and the question stops mattering.
- • Use conditional writes on a version instead of a lock, so a stale actor loses by construction without any lease machinery.
- • Make the work re-runnable and cheap, so a wrong eviction costs duplicated effort rather than duplicated effect.
- • Reduce pause frequency at the source — smaller heaps, appropriate CPU limits, non-blocking I/O — which does not make the design safe but does make the window rare.
Crashed or just slow — and why the answer does not matter
// not sufficient: the pause can land between the check and the write
if (lease.stillMine()) { // <- a 14s pause can begin right here
storage.write(key, value) // and this arrives after eviction
}
// sufficient: the check happens at the resource, atomically with the effect
storage.write(key, value, { fence: myToken })
// storage rejects any fence below the highest it has ever seenWhat people believe, and what is true
A long enough timeout removes the ambiguity.
It makes the wrong answer rarer and the detection slower. No finite timeout is an upper bound on delay in an asynchronous network.
Checking the lease before acting is sufficient.
The pause can land between the check and the action. The check must be atomic with the effect, which means it must happen at the resource.
Modern low-pause garbage collectors solve this.
They reduce one cause. Swap, CPU throttling, VM suspension and blocking I/O all remain, and none of them is under the runtime’s control.
If the node is really dead, our design works.
The design must work when the node is *not* dead and you concluded it was. That is the case worth testing, and it is the one that is rarely tested.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
You cannot tell a crashed node from a slow one — the evidence is identical. So do not build anything that depends on telling them apart; build so that guessing wrong is harmless.
Practical
Add a pause detector to every process. Give every exclusive action a monotonic token and have the resource — the database, the object store, the queue — reject stale tokens. Never check a lease in the client and then act; the pause goes in the gap. Then test it by sending SIGSTOP to a worker and confirming its later writes are refused.
Advanced
This undecidability is the engine of the FLP impossibility result: deterministic consensus cannot be guaranteed in an asynchronous system with even one crash failure, precisely because no protocol can distinguish a crashed participant from a slow one and must therefore either wait forever or risk deciding without it. Production systems escape by assuming partial synchrony — safety unconditional, liveness only during good periods — or by randomisation, which gives termination with probability one. What they never do is solve the detection problem, because it is not solvable. Every real design in this space is a way of not needing the answer.
Apply it
- 🔧 Take a job runner that uses a distributed lock and make it safe against a 60-second pause. Then demonstrate it by pausing the process and showing the late write is rejected.
- 💬 Why is telling a crashed node from a slow one undecidable rather than merely difficult?
- 💬 A worker checks that it still holds the lock, then writes. What is wrong with this?
- 💬 Name four causes of a multi-second pause in a process that is not crashed.