Backup Operations
Schedule, retention, encryption, access and verification — the six properties that decide whether a backup is protection or a green checkmark.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What has to be true of a backup before it counts as protection rather than a scheduled job that exits zero?
Data is the one part of production you cannot rebuild from source. Everything else — instances, images, configuration, DNS — can be recreated from a repository. A deleted table cannot.
Enable the managed backup option, take a nightly snapshot, watch the job status page, and move on. The provider handles durability, so the data is safe.
The job status only proves the backup process exited successfully. It does not prove the bytes are readable, the archive is complete, or the restore path works.
- The job status only proves the backup process exited successfully. It does not prove the bytes are readable, the archive is complete, or the restore path works.
- Nightly-only means the worst case loss is a full day of writes. Nobody agreed to that; it was inherited from a default.
- Retention shorter than detection time is the most common quiet failure: logical corruption is often noticed days later, by which time every retained copy already contains it.
- Backups stored in the same account, project or region as the primary share its failure and its compromise. An attacker or a mistaken automation with access to production usually has access to the backups too.
- Encryption without a key that survives the disaster produces an archive nobody can open, which is durability without recoverability.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A backup is a consistent point-in-time copy of state, plus enough metadata to interpret it: schema version, engine version, encryption key reference and the exact instant it represents.
- Consistency is the hard part. A file-level copy of a running database taken without coordinating with the engine can capture a page mid-write; engines solve this with a snapshot plus the write-ahead log needed to reach a consistent point (Write-Ahead Logging in Database Engineering).
- Two families do different jobs. A full copy (snapshot or logical dump) gives you a discrete point you can restore to. Continuous archival of the log stream lets you replay forward to an arbitrary instant, which is what point-in-time recovery means.
- The recovery point is set by the newest usable copy, not by the newest copy. A backup taken after corruption began is storage, not protection.
Six properties, and how each one fails silently
A backup configuration is six decisions, and each has a failure mode that leaves the dashboard green. Reading them as a checklist is the point: the job status covers exactly one of the six.
Notice that only the last row is evidence. The first five are intentions until a restore proves them.
| Property | The question it answers | How it fails with a green job |
|---|---|---|
| Interval | How much recent work can we lose? | Set from a tool default, never mapped to a tolerable loss |
| Retention | How far back can we go? | Window shorter than the time it takes to notice logical corruption |
| Isolation | Does the backup survive what killed the primary? | Same account, same region, same credential, same blast radius |
| Encryption | Who can read it if it leaks? | Key lives only in the environment that was lost |
| Access | Who can delete it? | Production automation can expire or overwrite copies |
| Verification | Does it restore into a working system? | Nothing checks; job exit status is treated as proof |
Which mechanism produces the copy
These are not tiers of quality. They answer different questions, they fail differently, and most production systems run two of them together — a periodic full copy to bound restore work, and a continuous log stream to bound data loss.
What produces the copy, and what does that choice buy?
when The engine or provider can take a consistent block-level snapshot of the volume; the dataset is large enough that a logical dump is impractical.
cost Restores into the same engine version and often the same provider only. Opaque — you cannot inspect or partially extract without restoring first.
when You want a portable, inspectable, engine-readable copy — for migration between versions or extracting a single table.
cost Time to take and to load scales with data volume and index rebuild, and both scale badly. Taking one on a busy primary competes with production for I/O.
when Losing more than the last few moments of writes is unacceptable, and the engine can ship its write-ahead log continuously.
cost A permanent write path that can fail or fill. Recovery requires a base copy plus an unbroken log chain; one missing segment truncates how far forward you can replay.
when You need fast failover for infrastructure loss.
cost Not a backup at all for logical damage — it applies the destructive change too. Useful alongside backups, never instead of them.
The verifications that catch a dead backup before you need it
Structural verification is cheap and runs per backup. It cannot prove the data is usable — only a restore drill does that — but it catches the majority of dead-backup cases early, when there is still a good copy behind them.
Every check below has the same shape: assert a property of the newest copy, and page on the assertion rather than on the job.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Backup runs against a read replica | Restored data is missing recent writes | Replication had stopped or lagged badly; the copy is consistent but stale | Assert replication lag as a precondition of the backup job and record the source instant in the manifest |
| Infrastructure rebuilt by a change to automation | No backups for days; nothing alerted | The alert fired on job failure, and no job existed to fail | Alert on backup age, from outside the system that takes them |
| Bad migration deletes a column's data | Every retained copy contains the damage | Retention window shorter than detection time | Extend retention past realistic detection latency; add an immutable long-horizon copy (Partial and Logical Data Recovery) |
| Production credential compromised or over-scoped automation runs | Primary and backups deleted together | Backups reachable from the production identity | Separate account, object lock, and delete permission held by an identity production does not have (Least Privilege in Production) |
| Engine upgraded | Older snapshots refuse to restore | Snapshot format is tied to the engine version | Record engine version in the manifest and re-drill after every major upgrade |
11. freshness age(newest usable backup) < agreed interval -> catches a schedule that stopped22. presence manifest lists every expected file / segment -> catches a partial upload33. continuity log/WAL segment chain has no gap since last full -> catches a broken PITR window44. size total bytes within a normal band vs recent runs -> catches an empty or truncated dump55. readable archive opens and the header parses with the6 production key path, from a machine that is not7 the primary -> catches a key nobody can use86. isolation a copy exists outside the primary account/region -> catches shared blast radius9 10None of the six proves the data restores into a working application.11That claim requires a restore drill.Check 5 fails far more often than people expect, and it is the one that is only discovered during an incident if you do not run it routinely.
How to do it properly
Most important first.
- Decide the interval from the tolerable loss, not from the tool default, and write the reasoning down next to the objective (RTO and RPO).
- Keep continuous log archival alongside periodic full copies wherever the engine supports it. Fulls bound restore time; the log stream bounds data loss.
- Store at least one copy outside the primary's failure domain and outside its access domain — a different account or project, ideally a different region.
- Make deletion hard: object lock, immutability windows or a separate credential for expiry. Ransomware and runaway cleanup scripts both target backups first.
- Encrypt at rest, and record where the key lives and who can use it during an outage. Test the key path, not just the cipher (When Secrets Fail).
- Verify structurally after every backup — the archive opens, the manifest is complete, the size is in family with yesterday — and verify by full restore on a schedule (Restore Drills).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains an unusable backup once you need it. That asymmetry — cheap to get right in advance, impossible to fix afterwards — is the whole argument for the module.
What can go wrong
- The backup succeeds against a replica that stopped replicating, so it is a perfect copy of stale data.
- The schedule quietly stops after an infrastructure change and nothing pages, because the alert is on failures and there are none.
- Retention expires the last good copy before the corruption is detected.
- The archive is encrypted with a key stored only in the environment that was lost.
- "Replication is a backup." Replication copies your mistakes at replication speed. A dropped table arrives on the replica immediately (Replication and Read Scaling in Database Engineering).
- "The provider guarantees eleven nines of durability, so we are safe." Durability protects against media loss. It does nothing about a bad migration, a bad DELETE or a compromised credential.
- "We back up the database, so we can recover the system." The system is the database plus object storage, plus configuration, plus secrets, plus the schema the code expects.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The most recent restore drill is recorded with a date, an operator, the data volume and the outcome (Restore Drills).
- A structural verification runs per backup and reports what it checked, not just that it ran.
- You can name, without looking, where a copy lives that a compromised production credential cannot reach.
- Changing backup configuration is itself a production change: it can silently reduce coverage. Keep the previous schedule and retention running until the new one has produced a verified restore.
- Never shorten retention and delete in the same change. Shorten the policy, wait out the old window, then remove.
- Automate: scheduling, log archival, structural verification, freshness alerting, replication of copies to a second domain, and expiry within the agreed window.
- Keep human: the decision to restore, the choice of recovery point, and any deletion of backup data outside the automated policy.
- Shorter intervals and longer retention cost storage and egress continuously to reduce a loss that may never happen. That trade should be made explicitly per dataset, not once for everything.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- DATABASE-SPECIFICPostgreSQL takes a base backup plus archived WAL for point-in-time recovery; MySQL/InnoDB uses a physical copy plus binlogs; MongoDB uses the oplog. The three differ in what a "consistent snapshot" costs on a busy primary and in whether logical dumps are viable at size — a dump that is fine for a small schema is unusable for a large one.
- CLOUD-SPECIFICManaged database snapshots are usually incremental, tied to the provider account, and restore into a *new* instance rather than over the existing one. Cross-account and cross-region copies are separate features you must enable; without them, losing account access loses the backups.
- GENERALThe six properties — interval, retention, isolation, encryption, access control, verification — apply to any datastore including object storage and message brokers, not only relational databases.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — what a consistent snapshot means across more than one node, and why a per-shard backup set is not automatically a consistent set.