The question this answers
The write returned 200 and then the machine died. Is the data still there?
A write is durable only against the failures its acknowledgement rule actually covers. fsync-then-ack survives a process crash and a power loss on that machine; it does not survive that machine ceasing to exist. Surviving the loss of a machine requires acknowledgement after the data is on a second one, and surviving the loss of a zone requires that second machine to be in another zone.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows that its own write reached the point in its own stack that it chose to wait for — a buffer, a page cache, an fsync return. It knows which peers have *told it* they have the data. It does not know whether a peer that acknowledged has since lost it, and it never knows whether its own disk actually persisted what fsync claimed. "Acknowledged by B" is knowledge. "Durable on B" is an inference, and the gap between them is where this class of data loss lives.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Three claims that get spoken as one
The word "durable" is doing at least three jobs in normal engineering conversation, and merging them is how teams end up genuinely surprised by data loss:
Acknowledged means the client got an answer. That is a statement about a message, not about storage. A system can acknowledge from a memory buffer, and many deliberately do — it is the fastest possible answer and it is the correct choice for some data.
Durable means it survives a crash of the process and a loss of power *on that one machine*. This is what fsync buys and what a write-ahead log makes efficient. The Database domain owns how that is implemented; what matters here is its scope, which is exactly one machine.
Replicated means a copy exists on another machine, and — if you want zone-failure survival — in another failure domain. This is the only one of the three that survives a machine being terminated, a disk being physically destroyed, or a rack going away.
The critical case, and the one worth memorising: a write acknowledged after a local fsync but before replication is durable on one machine and lost if that machine is gone. The fsync did its job perfectly. The data is on a platter. The platter is in a machine that no longer exists. This configuration is extremely common — it is the default in more systems than people expect — and it produces data loss with no error, no alert, and a completely healthy-looking dashboard.
| Ack rule | Process crash | Power loss on that machine | Machine destroyed | Zone lost |
|---|---|---|---|---|
| Ack from memory bufferprotocol | No | No | No | No |
| Ack after write() to page cacheprotocol | Yes | No | No | No |
| Ack after local fsyncassumption | Yes | Yes | No | No |
| Ack after 2 of 3 replicas, same zoneassumption | Yes | Yes | Yes | No |
| Ack after 2 of 3 replicas across zonesassumption | Yes | Yes | Yes | Yes |
| Ack after async replication startstypical | Yes | Yes | Only if the copy arrived first | No |
The window nobody measures
Between "acknowledged" and "replicated" there is a window, and its width is a real number you could measure and almost certainly do not. With Asynchronous Replication: The Loss Window You Chose, every write spends some time existing on exactly one machine. If that machine is lost during the window, the write is gone — and the client was told it succeeded.
This is the same quantity the Performance domain calls replication lag, seen from the correctness side rather than the staleness side. Staleness is a reader’s problem: a follower serves an old value. Loss is a writer’s problem: the value was never anywhere else. The same lag metric predicts both, which is why it deserves an alert threshold derived from your recovery point objective rather than from a default.
The spacetime picture below is the whole lesson in one diagram. Note what the client observes at every step: a success, and then nothing unusual, ever. The absence of the data is discovered later by a reader who was not there.
What fsync does and does not promise
Below the replication question sits an older and dirtier one. fsync asks the operating system to push a file’s data out of the page cache and onto stable storage, and to not return until it is there. That contract has been broken by more layers than anyone would like: drives with volatile write caches that report completion early, virtualised block devices that buffer, and — historically — file systems and drivers whose error handling on a failed writeback lost the error entirely, so a later fsync returned success for data that was never written.
The practical stance is not paranoia; it is scope. Treat single-machine durability as probably correct and not independently verifiable by you, and get your real durability from replication across machines, which you can verify. This is the same reasoning that makes Redundancy Is Not Resilience a real distinction: adding a second machine converts an unverifiable claim about one device into an observable property of a system.
It also explains a design choice that looks like cheating. Some systems acknowledge from memory on the primary while requiring the write to be in memory on two other machines. That configuration survives any single machine failure — including power loss, because the other two machines did not lose power — while never waiting for a disk at all. Whether it survives a *correlated* power event is exactly the Correlated Failure: The Independence Assumption Is Usually False question, and the answer depends entirely on whether those machines share a power domain.
Postgres synchronous_commit = off → ack before WAL fsync; a crash loses recent commits
synchronous_commit = local → local fsync only; machine loss loses them
synchronous_standby_names = ... → ack after a standby has the WAL
Kafka acks=1 → leader's page cache; leader loss can lose the batch
acks=all + min.insync.replicas=2 → two brokers hold it before the ack
Cloud disk fsync returns → in the volume service, not necessarily on a platter
(durability is the volume's replication, not your fsync)Recovery is where the difference becomes visible
A failover is the moment the acknowledgement rule is audited. The new primary has whatever it had; anything acknowledged but not replicated to it is now simply absent, and nothing in the system is aware that it is absent. There is no error to log, because from every surviving component’s point of view nothing went wrong.
That is why the honest recovery procedure includes a reconciliation step against an external record, not just a promotion. If the writes came through a queue, replay from the last durable offset. If callers keep their own record of what they submitted, compare. If neither exists, you cannot know what you lost — and "we cannot enumerate the lost writes" is the actual operational cost of an asynchronous acknowledgement rule, more than the loss itself.
The forgotten half is the old primary. If it comes back — a network partition healing, an instance restarting — it holds writes that the world has since decided never happened. Merging them in is usually wrong: they may conflict with writes accepted after the failover. This is why a rejoining node normally truncates to the new leader’s log rather than contributing, and why Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely exist to stop the old primary accepting anything in the interim.
- A failover audits your acknowledgement rule; nothing before it will.
- Lost writes generate no errors, so detection must come from an external record, not from the system itself.
- A returning old primary holds writes the world has discarded — reconciling them is a conflict-resolution problem, not a recovery step.
- Recovery point objective is the honest name for "how much acknowledged data are we willing to lose"; pick it deliberately.
Key points
- Acknowledged, durable and replicated are three different claims with three different scopes.
- fsync-then-ack is durable on exactly one machine and does not survive that machine being gone.
- The window between acknowledgement and replication is measurable, and its width is your real recovery point objective.
- Lost writes produce no error anywhere, which is why they are found by a reader days later rather than by an alert.
- Failover is the audit of the acknowledgement rule; if you have not chosen it deliberately, failover is when you find out what it was.
- You cannot verify your own disk’s durability, but you can verify that a second machine has the data.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The write reaches the primary and enters a write-ahead log buffer in memory.
- • The buffer is written to the file system — which by default means the page cache, not the disk.
- • Optionally, fsync is called, and the primary waits for the storage stack to claim persistence.
- • Optionally, the record is sent to replicas, and the primary waits for some number of them to acknowledge receipt (or their own fsync).
- • The primary answers the client. Which of the preceding steps it waited for *is* the durability guarantee — nothing else in the system defines it.
- • Any step the primary did not wait for continues asynchronously, creating a window in which the acknowledged write exists in fewer places than the client believes.
- • The machine loses power between the acknowledgement and the fsync.
- • The machine is terminated between the fsync and the replication.
- • A replica acknowledges receipt into memory and then crashes before persisting it.
- • The disk reports a successful fsync for data still sitting in a volatile cache.
- • A whole zone is lost, taking every replica that was placed in it.
- • A failover promotes a replica that is behind, and the missing writes become unrecoverable at that instant.
- • The invisible gap: after an instance replacement, a set of records that returned 200 are simply not present. Error rates, latency and replication metrics all look normal for the entire incident, because no component failed.
- • Ledger mismatch: a downstream consumer that recorded every submitted request finds N records the database does not have. The consumer is the only reason anyone knows.
- • The rejoining ghost: the old primary comes back and reintroduces rows that were superseded after failover, producing state that matches neither the old nor the new timeline.
- • Silent RPO drift: replication lag creeps from 200 ms to 40 s over months as write volume grows. Nothing alerts, because no threshold was ever tied to a data-loss objective — until a failover loses forty seconds of writes.
- • fsync stall masquerading as an outage: latency spikes and the service looks hung, because durability was configured strictly and the disk is slow. The operator relaxes the setting to fix the latency, silently trading durability for speed under pressure.
- • Durability that survives machine loss requires waiting for another machine — one round trip added to every write. That is the price, and it is the same price Synchronous Replication: Paying Latency for a Durability Guarantee charges.
- • Waiting for more replicas raises durability and lowers availability: if the required number is unreachable, writes stop. This is the availability face of Coordination Couples Availability.
- • The commonly chosen middle is "acknowledge after any two of three", which tolerates one slow or dead replica while still guaranteeing the write outlives any single machine.
- • Waiting for a replica in another region adds the round trip that The One Number You Cannot Optimise sets a floor on — often the reason cross-region synchronous durability is rejected.
- • Everything acknowledged under the rule you chose survives the failures that rule covers — exactly those, and no others.
- • Writes in the acknowledgement-to-replication window are lost with no trace and no error.
- • Reads continue to be served correctly from whatever survived; the system is consistent with a history that quietly omits the lost writes.
- • The system cannot tell you what it lost, because it never knew those writes were at risk.
- • Detect: compare against an external record — a queue offset, a caller-side log, an upstream ledger. The store cannot detect its own gaps.
- • Contain: fence the old primary so it cannot accept writes into a timeline the cluster has abandoned.
- • Recover: replay from the most upstream durable source you have. This is precisely why keeping the ingest queue’s retention longer than your worst failover is cheap insurance.
- • Reconcile: for writes with external side effects that were lost, re-drive them idempotently rather than reconstructing state by hand.
- • Verify: after failover, sample recent acknowledged writes and confirm they are present on the new primary. Then publish the measured gap, so the next RPO conversation uses a number.
- • Replication lag in bytes *and* in seconds — bytes tells you how much is at risk, seconds tells you how far behind a reader is.
- • Count of in-sync replicas per partition or shard, alerted below the number your acknowledgement rule assumes.
- • fsync latency distribution on the write path; a p99 in the hundreds of milliseconds is the pressure that makes people weaken durability settings.
- • The acknowledgement rule itself, exported as a metric or a config-drift check — this is the number that decides everything and it is usually only found in a file.
- • Post-failover gap measurement as a standing procedure, so RPO is an observed value rather than an aspiration.
- • Any data whose loss is not recoverable from upstream: payments, orders, audit records, anything a user typed once.
- • Systems where the recovery point objective is contractual, because the acknowledgement rule is the only thing that actually implements it.
- • Post-incident analysis: this vocabulary turns "we lost some data" into a specific, fixable configuration decision.
- • For data that is trivially reproducible — derived caches, recomputable aggregates, metrics samples — strict durability buys latency cost for nothing. Acknowledge from memory and move on.
- • For very high write rates where the workload is a firehose and losing the last second is genuinely acceptable, synchronous durability can halve throughput to protect data nobody would miss.
- • Applying the strictest rule uniformly makes every write pay for the most valuable write, which is usually the wrong global optimum.
- • Put a durable queue in front and treat it as the source of truth, so the database can acknowledge fast and anything lost is replayable. This moves the durability requirement to one system instead of every system.
- • Choose durability per class of data rather than per cluster: strict for the ledger, relaxed for the telemetry.
- • Accept a defined recovery point and buy back the risk with backups plus point-in-time recovery, which is far cheaper than synchronous cross-region replication.
- • Let the client hold the write until it is confirmed downstream — an outbox on the caller’s side turns an unverifiable ack into a reconcilable record.
The write returned 200 and the machine died. Is the data still there?
Postgres synchronous_commit = off → ack before WAL fsync; a crash loses recent commits
synchronous_commit = local → local fsync only; machine loss loses them
synchronous_standby_names = ... → ack after a standby has the WAL
Kafka acks=1 → leader's page cache; leader loss can lose the batch
acks=all + min.insync.replicas=2 → two brokers hold it before the ack
Cloud disk fsync returns → in the volume service, not necessarily on a platter
(durability is the volume's replication, not your fsync)
you have selected: Postgres synchronous_commit = local| Process crash | Power loss on that machine | Machine destroyed | Zone lost | |
|---|---|---|---|---|
| Ack from memory bufferprotocol | No | No | No | No |
| Ack after write() to page cacheprotocol | Yes | No | No | No |
| Ack after local fsyncassumption | Yes | Yes | No | No |
| Ack after async replication startstypical | Yes | Yes | Only if the copy arrived first | No |
| Ack after 2 of 3 replicas, same zoneassumption | Yes | Yes | Yes | No |
| Ack after 2 of 3 replicas across zonesassumption | Yes | Yes | Yes | Yes |
What people believe, and what is true
The write returned 200, so it is safe.
It is as safe as the acknowledgement rule made it, which may mean "in a buffer on one machine". The status code carries no information about how many places the data exists in.
fsync makes the data durable.
It makes it durable on that machine. If the machine is terminated, deleted or physically destroyed, fsync bought you nothing.
Three replicas means three copies at all times.
It means three copies eventually. With asynchronous replication there is always a window in which the count is one.
A backup covers this.
A backup bounds how much you lose to the time since the last one, and restoring is measured in hours. It is a floor under the disaster, not a substitute for an acknowledgement rule.
We would see an error if we lost writes.
This is the defining property of the failure: no component failed, so no component logged anything. Detection requires comparing against something outside the system.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Acknowledged means you got an answer. Durable means it survives a crash of one machine. Replicated means it survives that machine disappearing. Know which one your 200 means.
Practical
Find the acknowledgement setting for every store you own and write down what it survives. Alert on in-sync replica count and on replication lag against a threshold derived from your RPO. Keep the upstream queue’s retention longer than your worst-case failover, so replay is possible. Measure the gap after every failover and publish it.
Advanced
Durability is not a property of storage; it is a property of the *acknowledgement protocol*, and the only lever is which step the ack waits for. This makes it a direct trade against availability: waiting for more machines means fewer machine failures you can tolerate before writes stop. That is Quorums: What R + W > N Does and Does Not Buy seen from the durability side — the same overlap argument, with data loss rather than stale reads as the thing being prevented.
Internals
Down the stack the guarantee gets muddier. A group commit batches several transactions into one fsync, so the durability boundary is a batch, not a transaction. A replica that acknowledges from its receive buffer has moved the same ambiguity one machine sideways. A cloud block volume is itself a replicated service, so your fsync is a network round trip to a system with its own acknowledgement rule that you did not choose and cannot see. At the bottom, the honest statement is that you have layered several probabilistic claims and can verify only the ones spanning machines you can query.
Apply it
- 💬 A write returned 200 and the instance was replaced ten seconds later. Under what settings is the write still there?
- 💬 How would you find out how much data a failover lost?
- 💬 Your database has three replicas and you still lost acknowledged writes. Explain how.
- 💬 When is acknowledging from memory the right choice?