Recovered State Is a Checkpoint Plus the Log After It

Consider a counter at 4,000,001. You can persist the number, or you can persist four million increments. State is compact and fast to load, but it must be written atomically and it destroys history. A log is cheap to append — it is a sequential write, the one thing every storage device is good at — but replaying four million records to learn one number is absurd.

The resolution is the oldest structure in systems: checkpoint the state occasionally, log every change continuously, and recover by loading the newest checkpoint and replaying the log records after it. The checkpoint bounds replay time. The log bounds data loss. Neither alone does both.

The vocabulary changes by layer and the structure does not. A database calls it a write-ahead log plus a checkpoint. A consensus implementation calls it a snapshot plus the The Raft Log: Commit Index, Divergence and Reconciliation suffix. A stream processor calls it state-backend checkpointing plus offsets. An event-sourced application calls it a snapshot plus the event stream. If you have understood it once, you have understood all of them, and the Database domain owns the mechanics of doing it well on one machine.

The whole scheme rests on one ordering constraint: the log record must be durable before the change it describes is applied to the main structure. That is the "write-ahead" in write-ahead logging, and reversing it breaks recovery — a crash between applying and logging produces a state whose history nobody has.

From that constraint two useful properties follow. Recovery is *idempotent*: replaying a record whose effect is already present must be harmless, which is why records are keyed by a monotonically increasing position and application is conditional on that position. And recovery is *bounded*: you only ever replay from the checkpoint forward, so replay time is a function of checkpoint frequency, not of the system’s age.

The Database domain owns how this is implemented — record formats, group commit, fuzzy checkpoints, the interaction with the buffer pool. What belongs here is the observation that the constraint is about order and durability of the record relative to the change, and that this is a property you can state and check for any system that claims to be recoverable, at any layer.

Here is the observation this lesson exists for. Recovery and replication are the same operation with a different destination. Recovery replays the log into the same node after a crash. Replication ships the log to a different node and replays it there. In both cases the receiving state machine starts from a known position and applies an ordered sequence of deterministic changes.

That is not an analogy — it is literally how the systems are built. Postgres streams the WAL to standbys. Raft *is* a replicated log whose commit rule decides which prefix everyone must apply. Kafka replicas fetch the leader’s log and apply it. When a new replica joins with no state at all, it does not get a special protocol: it gets a checkpoint transfer followed by the log from that checkpoint’s position, which is the recovery procedure with a network hop inserted.

This explains why log-based systems dominate this space. Choosing a log as your durability mechanism gets you replication, point-in-time recovery, and change data capture from the same artefact, because all three are "read the ordered record of changes from some position". Choosing to persist state only gets you none of them. It is also why The Log Is Not a Queue and A Topic Is Not One Log: Ordering Lives Inside a Partition keep reappearing in this domain: the log is the primitive that makes several otherwise-separate problems into one.

The catch is determinism. Replay reproduces the original state only if applying a record is a pure function of the record and the prior state. A record saying "set expires_at to now() + 1 hour" replays to a different value; the log must therefore record the *computed result*, not the instruction that computes it. Any non-determinism — wall-clock reads, random values, external calls — has to be resolved at record time and written down, or the recovered state diverges from the state you lost.

Checkpoint frequency is a straight trade with exactly two sides. Frequent checkpoints mean short replay and therefore short recovery, at the cost of steady I/O and, for large state, a visible latency bump while the checkpoint is taken. Infrequent checkpoints are cheap in steady state and expensive exactly once — during an incident, when you are watching a node replay for forty minutes and can do nothing about it.

The asymmetry is what people get wrong. Checkpoint cost is paid continuously and is easy to see on a graph; replay cost is paid rarely and only when you can least afford it. Teams therefore drift toward long intervals, because the graph rewards it and nothing punishes it until an incident. The number to hold is not the checkpoint interval but the resulting worst-case replay time, and that number should be measured rather than estimated — by actually restarting a node with a full log suffix and timing it.

The second trap is log retention. Truncating the log before the checkpoint that covers it makes the state unrecoverable — an operational mistake with no gradual warning, only a cliff. Correspondingly, retaining log beyond the oldest checkpoint any *replica* might still need is what lets a lagging replica catch up without a full state transfer; truncate too eagerly and a replica that was thirty seconds behind now needs a fourteen-gigabyte snapshot copy.