Internals · Buffer Poolwrite pathUPDATEdirty pageWAL recordpage LSN

Follow a Write Through the Engine

UPDATE accounts SET balance = balance - 100 WHERE id = 42 finds the row like a read, then logs the change, modifies the page in memory, touches an index only if an indexed column changed, marks the page dirty, fsyncs the log at COMMIT — and writes the data page minutes later. The gap between COMMIT and that flush is where durability is decided.

▶ InteractiveTry queriesInterview question
Progress

Why this exists

The mechanism as the answer to a problem — read this before the name.

  1. Problem

    A read leaves the pages as it found them. A write must change bytes on a page that lives in RAM, keep the indexes consistent, and make the change survive — while other transactions read the same page.

  2. Naive solution

    Find the row, change the bytes in the frame, mark the frame dirty, tell the client OK. The buffer pool will write the page eventually.

  3. Why it breaks

    If power fails between OK and "eventually", the change exists nowhere. If the page is written mid-transaction and the transaction then aborts, storage holds a change that never committed. If an indexed column changed and the index was not updated, the index lies.

  4. Better idea

    Describe the change in a log before making it, stamp the page with the log position, update every affected index the same way, and make COMMIT mean "the log is on disk" — leave the page write for later.

  5. Internal mechanism

    Find the record (index descent) → load and pin the page → assign the transaction id and lock the row → append a WAL record (before/after image, LSN) → modify the page in memory → update indexes whose key changed → mark pages dirty with the page LSN → COMMIT: append and fsync the commit record → flush pages later.

  6. Trade-offs

    Every write becomes at least two writes — log now, page later — and every index on a changed column adds another modified page and another log record. The design moves the cost from random page writes to one sequential log write per commit, which is the trade worth making.

  7. Real database

    PostgreSQL: a new tuple version on the same page (HOT) if no indexed column changed and space allows; xmax on the old version; one WAL record per modified page. InnoDB: in-place update of the clustered row, the old image into the undo log, secondary indexes updated through the change buffer.

Choose your depth

The same mechanism at four altitudes. Start where you are; come back deeper.

A write changes memory now and disk later

An UPDATE starts exactly like a SELECT: it has to find the row, and that means walking the index down to the heap page. Then it changes a few bytes inside the page in memory. The page is now different from the copy on disk — dirty — and the disk copy will be brought up to date later, in the background.

Because the disk copy lags, the engine writes a short note describing the change to a log before it touches the page. COMMIT means "that note is safely on disk". The data page itself may not be written for minutes, and that is fine, because the note is enough to redo the change if the machine dies.

A write starts as a read

The planner treats WHERE id = 42 exactly as it would for a SELECT: an index scan on accounts_pkey — root, internal, leaf — yields tuple id (2093, 7). The heap page is requested from the buffer pool and pinned. So far nothing has been modified, and the cost so far is the same four page requests as the read path in Follow a Read Through the Engine. An UPDATE with no usable index scans the table to find its rows, and it holds locks while it does so; the slowest writes are usually slow reads.

The transaction supplies the identity for everything that follows: transaction id 7412 will be written into the log records, into the tuple header (PostgreSQL xmax on the old version, xmin on the new), and into the lock manager. A concurrent UPDATE of the same row sees the row locked and waits — the beginning of everything in The Lock Manager.

Log first, then page

Before a byte of the page changes, the backend appends a record to the write-ahead log: transaction id, relation, page number, slot, the old balance 500, the new balance 400 — around 80 bytes. The record receives an LSN, its byte offset in the log stream, which will be stamped on the page. This ordering is the entire meaning of "write-ahead": the description of a change reaches the log before the change can reach storage. The record is in the log *buffer* in memory for now; nothing has been fsynced. What matters is the order, and the order is enforced by the page LSN: no page may be written to storage while the log is not durable up to its LSN.

Then the page is modified. PostgreSQL places a new tuple version in the page's free space (new xmin = 7412, balance 400) and sets xmax = 7412 on the old version; the old version stays for readers whose snapshot predates the transaction — this is the mechanism of MVCC Internals: Version Chains and Snapshots. InnoDB overwrites the row in place and copies the old image into an undo log page, which is itself logged. Either way the change is a few bytes moved inside an 8 KB frame: the cheapest step in the whole write.

The WAL record for this update (AtlasDB V7 layout, PostgreSQL-like)
LSN 5 812 331  ┌ length 84 · txid 7412 · prev-LSN 5 812 247 · CRC ┐
               │ rmgr HEAP · op UPDATE                              │
               │ relation accounts (16 421) · block 2093            │
               │ old slot 7  → xmax = 7412                          │
               │ new slot 12 → xmin = 7412, balance 400             │
               └ (before-image 500 kept for undo-capable engines)   ┘

Mark dirty, stamp the LSN

The frame's dirty bit is set and the page header's LSN field becomes 5 812 331. From here the page has two futures. The background writer or the next checkpoint will write it — after first ensuring the log is durable through 5 812 331 — and clear the dirty bit. Or the frame will be chosen as a victim by some other backend's miss, which then has to perform that write on its own critical path. Until one of those happens, storage holds balance 500 and the page in RAM holds 400, and the only durable evidence of 400 — once COMMIT has done its fsync — is the log record.

The page LSN does double duty. It enforces the write-ahead rule going forward, and during recovery it tells redo whether a page already contains a change (page LSN ≥ record LSN → skip) — which is what makes recovery safe to repeat. Both are covered in Crash Recovery.

Heap page 2093 in the pool after the modification (PostgreSQL-shaped)
┌ page header: LSN 5 812 331 · lower 76 · upper 6 840 · flags ─────────────┐
│ slots: … [7: off 7 112, len 64] … [12: off 6 840, len 64 (NEW)] …        │
│                                                                           │
│ free space ← shrank by 64 B                                               │
│ @6 840 tuple v2: xmin 7412, xmax 0     | id 42 | balance 400              │
│ @7 112 tuple v1: xmin 4 117 220, xmax 7412 | id 42 | balance 500  (old)    │
└ frame: dirty=1 · pin=1 (until the statement finishes) ────────────────────┘

Does the index change?

PostgreSQL implementation

The primary key did not change, so accounts_pkey still maps 42 → the row. But in PostgreSQL the row is now a *new tuple* at a new slot, and an index entry points at a slot. Updating every index on every UPDATE would make writes cost one page per index. The escape is the HOT update (heap-only tuple): if no indexed column changed and the new version fits on the same page, the old slot is turned into a redirect pointer to the new version, and every index continues to point at the old slot. The lookup follows the chain on the heap page; no index page is read or written. If balance were indexed, the index would need a new leaf entry (400 → new tid) and the old entry (500 → old tid) would remain until VACUUM removes it — a second B+ tree descent, a second dirty page, a second WAL record, and possibly a leaf split.

The practical rule: an UPDATE costs one modified page plus one per index whose key changed. Indexes on columns that change on every write (updated_at, last_seen, counters, status) multiply write cost and defeat HOT; a fillfactor of 80–90 leaves room for the new version on the same page and makes HOT possible at all.

  • HOT succeeds: no indexed column changed, and the page has room → indexes untouched, one dirty page.
  • HOT fails: an indexed column changed, or the page is full → every index gets a new entry pointing at the new tuple; the old entries stay until VACUUM.
  • pg_stat_user_tables.n_tup_hot_upd / n_tup_upd is the fraction of updates that took the cheap path.

InnoDB: in place, with undo and the change buffer

MySQL / InnoDB implementation

InnoDB updates the clustered-index row in place, so the row's identity (its primary key) does not move and secondary indexes are unaffected unless their column changed. The old row image goes to the undo log — needed for rollback and for MVCC readers — and both the row change and the undo write are described in the redo log. A changed secondary-index column produces a delete-mark of the old entry and an insert of the new one; if the secondary leaf page is not in the buffer pool, the change is recorded in the change buffer and merged when the page is next read, turning a random read-modify-write into a deferred one.

The redo record InnoDB writes is physiological — "on page P, apply this logical change to record R" — and small; the redo log is a fixed-size ring of files (innodb_log_file_size) that must be checkpointed before it wraps, which is why a burst of writes can stall on "log wait" when the checkpointer falls behind.

COMMIT, and the gap that decides durability

COMMIT appends a commit record for 7412, writes the log buffer through it, and calls fsync. When fsync returns the client is told OK. Count what is on disk at that moment: the log records (update, commit) — and nothing else. Heap page 2093 on storage still says 500. The dirty frame will be written later, perhaps in seconds, perhaps at the next checkpoint in five minutes.

So: the machine loses power one second after the client saw OK. The frame is gone. Storage says 500. Does the transaction survive? If the engine kept the log, yes — recovery reads the commit record, finds the update record, sees that page 2093's LSN on storage is older than 5 812 331, and rewrites balance 400 into the page. If the engine did not keep a log, the answer is no: the change existed only in RAM, the client holds a receipt for a transfer that never happened, and there is no evidence anywhere. That gap — committed but not flushed — is not a bug to close; it is the design, and closing it correctly is the subject of Write-Ahead Logging.

The write path: log before page, commit before flush
yesno (HOT)seconds…minutesindex scan → tidpin heap pageWAL record, LSNmodify frame, mark dirtyindexed column changed?update index leafCOMMIT: fsync log → OKflush page later
UserLLMAgentToolDataDecisionHumanGuardrail

Key points

  • A write begins with the read path: find the row through the index, pin the heap page.
  • Log first, then modify: the WAL record gets an LSN, the page is stamped with it, and the dirty frame may not be written before the log is durable to that LSN.
  • An index is updated only if an indexed column changed (InnoDB) or, in PostgreSQL, if the new tuple version cannot be a HOT update; every such index is another page, another record.
  • COMMIT fsyncs the log and returns; the data page is flushed later. At that moment the log is the only durable copy of the change.
  • Crash after COMMIT, before flush: with a log the change is redone; without one it is gone with a receipt in the client's hand.

Follow a write: UPDATE accounts

Follow a write: UPDATE accounts SET balance = balance - 100 WHERE id = 42
Find, load, log, modify, commit — and only later flush. Then press crash at any point and ask whether the transaction survives.
1UPDATE accounts SET balance = balance - 100 WHERE id = 42;
Find the record accounts_pkey → tid (2 093, 7)
What happens
The planner chose an index scan on `accounts_pkey`. Root → internal → leaf, exactly as in the read path, yields tuple id (2 093, 7): heap page 2 093, slot 7.
Why
A write begins as a read. An UPDATE that cannot use an index scans the table to find its row — and holds locks while it does.
Buffer pool
heap page 2 093: not loaded
WAL
(no record yet)
Storage
heap page 2 093: balance 500
WAL file: nothing for this txn
Pages read
3
Dirty pages
0
WAL bytes
0
fsyncs
0
Random page writes
0
On disk right now: nothing changes on disk.
Educational simulation — page sizes, costs and counters are modelled, not measured from a real engine.
1/9 · Find the record

Try it in the playground

When to use — and when not

Use it when
  • This log-then-page, commit-then-flush design fits every engine that wants both write-back caching and durable commits — which is every transactional database.
  • Reasoning about writes at this level fits when write latency or write amplification is the problem: count dirty pages, index touches and fsyncs per statement.
Avoid it when
  • It fits poorly for append-only, replay-from-source workloads (analytics ingestion, logs) where losing the last second of data is acceptable and the fsync per commit is pure overhead — those batch and relax synchronous_commit.
  • It is the wrong lens when writes are slow because the search is slow: fix the read path first.

Failure modes

  • An index on a column updated in every write (updated_at), defeating HOT and turning one dirty page into six.
  • Tables at fillfactor 100 with hot rows: no room for the new version on the page, so every update becomes a full index update.
  • Assuming COMMIT wrote the table: a backup of the data files without the WAL is a backup of a state that never existed.
  • Write bursts stalling on log space (InnoDB "log wait", PostgreSQL checkpoint storms) because dirty pages were allowed to accumulate for too long.

Where you meet this

Back up to the practical layer, and across to the rest of Engineer Atlas.