CachingGENERALDATABASE-SPECIFICSIMPLIFIED
Cache Invalidation
The database changed. How does the cache find out? Four answers, each with a different failure when it does not.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The question
When the underlying data changes, how does the cached copy learn about it — and what happens when that message is lost?
The requirement
An editor fixes a typo in a product title and expects to see it on the site. Not in an hour, and not after a deploy.
The obvious build
Delete the cache key in the same function that writes the row. The write and the delete sit next to each other, so they happen together.
Why it breaks
They do not happen together. The delete is a second network call to a second system with no transaction spanning both — the database can commit and the delete can fail, leaving a stale entry with a full TTL and no error visible to the user.
How it breaks in production
- They do not happen together. The delete is a second network call to a second system with no transaction spanning both — the database can commit and the delete can fail, leaving a stale entry with a full TTL and no error visible to the user.
- It only covers writes that go through this function. A bulk update, an admin script, a migration or a second service writing the same table invalidates nothing.
- One row appears in many cached shapes: the product entry, the category listing, the search result page, the homepage fragment, the API response for three different clients. Deleting one key leaves five stale.
- Delete-before-commit is worse than it looks: a concurrent reader repopulates from the pre-commit state between your delete and your commit, and the stale value survives the write entirely.
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale
What is actually happening
- The core problem is that the cache and the database are two systems with no shared transaction. Any invalidation is a second, independently-failing operation — this is the dual-write problem wearing a different hat (The Dual Write Problem).
- TTL is invalidation by giving up: you do not find out, you simply stop trusting the value after N seconds. It requires no coordination at all and is the only strategy that survives writers you do not know about (TTL and Expiry).
- Explicit invalidation deletes the key at the write site. Precise and immediate when it fires; silent when the write path does not go through your code.
- Event-driven invalidation decouples the two: the writer emits a
product.updatedevent and a consumer deletes the keys. It catches every writer that emits the event and gives you retries and a dead-letter queue for free — at the cost of an asynchronous staleness window (Writing Event Consumers). - Versioning / key rotation never invalidates anything. The version is part of the key, so a write makes every old key unreachable rather than deleting it.
product:42:v7becomesproduct:42:v8; the old entry ages out under eviction. This is the only strategy with no deletion step to fail. - These compose, and in production they usually must: explicit invalidation for the writes you control, events for the writes other services make, and a TTL underneath both as the backstop for everything you missed.
Four ways the cache finds out
These are not alternatives to choose between once. They are layers, and a mature system runs three of them at the same time: explicit invalidation for the writes it controls, events for the writes it does not, and a TTL underneath as the bound on everything that got missed.
Read the last column first. What each strategy does when it fails is the thing that decides whether you can put it in front of data that matters.
| Strategy | How the cache learns | Staleness window | When it fails, you get |
|---|---|---|---|
| TTL only | It does not — it stops trusting the value | Up to the full TTL, always | Nothing new. TTL cannot fail; it can only be too long (TTL and Expiry) |
| Explicit invalidation | The writer deletes the key after commit | Milliseconds, when it fires | A stale entry for a full TTL, with no error anywhere |
| Event-driven | A consumer deletes on product.updated | Consumer lag — normally small, unbounded under backlog | A retry, then a dead-letter entry you can see and replay (Dead-Letter Queues) |
| Versioning / key rotation | It does not — old keys become unreachable | Zero for the version bump itself | Orphaned entries consuming memory until eviction |
| Change-data-capture | A consumer reads the database's own change log | Replication lag (Replication Lag: Reads That Are Correct and Stale) | A stalled slot or growing lag, both of which are visible |
Order matters: commit, then invalidate
GENERALThe outbox pattern here is the same one as The Transactional Outbox; the only difference is that the downstream effect is a cache delete rather than a message to another service. Duplicate deletes are harmless, which makes invalidation an unusually forgiving consumer.
Both orderings fail. They fail differently, and one of the failures is recoverable while the other is not. If you invalidate first and then commit, a concurrent reader can repopulate the cache from the pre-commit state, and the stale value now has a full TTL ahead of it — the invalidation ran and achieved nothing. If you commit first and then invalidate, a failure leaves a stale entry too, but every subsequent invalidation, every TTL expiry and every retry fixes it.
The event-driven variant makes the ordering guarantee structural instead of a convention: the event is written in the same transaction as the row via an outbox, and the consumer cannot possibly run before the commit that produced it.
UserLLMAgentToolDataDecisionHumanGuardrail
What actually goes wrong
Stale-cache incidents are reported in a vocabulary that hides their cause — "the site is showing old data", "it fixed itself", "only some users see it". Each of those maps to a specific mechanism, and recognising the mapping is most of the debugging.
"It fixed itself" is the most diagnostic phrase in the list. It means a TTL expired, which means invalidation did not run and the backstop caught it.
Stale data, by symptom
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Editor saves, page still shows the old title | Stale until it "fixes itself" later | Invalidation failed or was never wired; the TTL is what eventually corrected it | Count invalidation failures; alert on the gap between attempted and succeeded |
| Some instances show new data, others old | Refresh flips between two values | In-process cache invalidated on one instance only (Local vs Distributed Cache) | Move to a shared cache, or broadcast invalidation, or shorten the TTL to the tolerable window |
| A bulk import runs | Latency spike and a database CPU spike minutes later | Mass invalidation, then every key missing at once | Jitter the invalidation, or repopulate in the background rather than deleting (Cache Stampede) |
| A user updates their profile and immediately sees the old one | Only the writer sees it; everyone else is fine | Invalidation is asynchronous and lost the race with the redirect | Read authoritatively on the writer's own path; do not rely on invalidation for read-your-writes |
| Data changed by a migration or an admin script | Stale for the full TTL, everywhere | The write path never went through the application code that invalidates | Event- or CDC-driven invalidation, which is the only kind that catches writers you did not write |
| A deploy changes the cached value shape | Fields silently undefined on some requests | Old entries deserialized as the new type | A version marker in the key or the envelope, so old entries read as misses |
How to build it
Most important first.
- Always set a TTL, even when you invalidate explicitly. The TTL is not the mechanism — it is the bound on how long a *missed* invalidation can hurt you. An entry with no TTL and a failed delete is stale until someone restarts the cache.
- Invalidate after the transaction commits, never inside it. Deleting before the commit lets a concurrent reader repopulate the pre-commit value, and the invalidation is wasted (Where the Transaction Boundary Goes).
- Derive keys from a single function so every writer and every reader agrees on the format. Then a "which keys does this row appear in" question has an answer someone can read.
- For a row that appears in many shapes, prefer a version key: store
product:42:ver -> 7and build derived keys aslisting:cat3:pv7. One increment makes every derived entry unreachable, without enumerating them. - Move to event-driven invalidation as soon as more than one writer exists. The moment a second service or a scheduled script writes the table, call-site invalidation is structurally incomplete (Event-Driven Backends).
- For anything that must be correct after a write from the same user, do not rely on invalidation timing at all — read the authoritative row on that path. Read-your-own-writes is a requirement, not a nice-to-have, and a cache does not provide it.
What can go wrong
Failure modes
- Commit succeeds, invalidation fails. The classic, and the reason a TTL is mandatory rather than optional.
- Invalidation succeeds, commit rolls back. The cache is now empty and the next read repopulates it correctly — the benign direction, which is why invalidate-after-commit is safe and invalidate-before is not.
- A key you forgot: the row is stale in a list, a search index or a rendered fragment nobody enumerated (Keeping a Search Index in Sync).
- Event consumer lag turns the staleness window from milliseconds into however far behind the consumer is (Queue Backlog).
- Mass invalidation as an availability event: a bulk update invalidates a large fraction of the cache at once and every subsequent request misses simultaneously (Cache Stampede).
- Multi-instance in-process caches: an explicit delete invalidates the local copy on *one* instance, and the other instances keep serving the old value (Local vs Distributed Cache).
What can race
- Delete-then-commit: a reader repopulates from the uncommitted state in the window between the delete and the commit, and the stale value outlives the write.
- Reader loads at T0, writer commits and deletes at T1, reader populates at T2 with the T0 value. The delete happened and the cache is stale anyway.
- Two writers, two invalidations, two repopulations, interleaved such that the older row is the one that lands (Backend Races).
- Version increment and derived-key populate interleaving: a reader computes a derived key from version 7, the writer moves to 8, and the reader writes an entry under the now-orphaned v7 key. Harmless — the entry is simply never read again. This is why versioning is the safest of the four.
Security
- Permission changes must invalidate cached authorization decisions. A revoked role that stays cached is an access-control failure with a TTL-shaped duration (Role-Based Access Control).
- Cached session or token state must be invalidated on logout and on password change, or "sign out everywhere" is a UI label with no server-side effect (Where Sessions Live).
- Do not expose cache-purge endpoints without authorization and rate limits. An attacker who can invalidate at will can empty your cache and push full read load onto the database on demand.
- Version-key rotation is safer than deletion for authorization data, because a failed delete leaves a permissive value readable while a failed increment simply leaves the old key unreferenced.
Misreads
- "Cache invalidation is hard because naming things is hard." The joke has cost the industry real understanding. It is hard because it is a distributed write to two systems with no shared transaction, and because one row appears in many derived shapes.
- "We invalidate on write, so the cache is consistent." It is consistent when the delete succeeds, from writers that go through your code, for keys someone remembered. Three conditions, each of which fails routinely.
- "A short TTL means we do not need invalidation." It means the *maximum* staleness is short. If a user must see their own write immediately, even one second is a bug report.
- "Write-through solves invalidation." It relocates it. The cache write and the database write are still two operations that can fail independently, and it does nothing for derived keys.
Operating it
How you see it in production
- Count invalidations attempted versus invalidations that succeeded. The gap is your stale-entry rate and almost nobody measures it.
- Track the age of served values — the time between populate and read — as a distribution. That distribution is your actual staleness, as opposed to the TTL you configured (Depth Is Not an Emergency; Age Is).
- For event-driven invalidation, consumer lag *is* the staleness window. Alert on it as a correctness signal, not a throughput one.
- Log key, reason and trigger on every invalidation, sampled. "Why did this go stale at 14:02" needs a trail, and reconstructing it from application logs after the fact is not possible.
- Watch for invalidation storms: a spike in deletes followed by a spike in origin queries is a bulk write in disguise (Deploys Are the First Suspect).
What changes at 10x and 100x
- Explicit invalidation scales with the number of *write sites*, which grows with the team. Event-driven invalidation scales with the number of *event types*, which grows with the domain — much more slowly.
- At 100x data, enumerating derived keys becomes impossible and version keys become the only workable strategy.
- Cross-region deployments turn invalidation into a replication problem: the delete has to reach every region's cache, and it arrives at different times (Multi-Region Deployment).
- In-process caches at 100 instances mean 100 invalidation targets. Either broadcast (pub/sub) or accept a TTL-bounded window; there is no third option.
What this costs
- TTL-only is the simplest thing that cannot silently fail, and it guarantees staleness for the whole window even when nothing changed.
- Explicit invalidation is immediate and precise, and it is only as complete as your knowledge of every write path — which decays as the codebase grows.
- Event-driven invalidation is complete and retryable, and it adds a broker, a consumer, a lag metric and a dead-letter queue to your operational surface (Dead-Letter Queues).
- Version keys never fail to invalidate, and they leave orphaned entries occupying memory until eviction reclaims them — you trade correctness risk for memory.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe four strategies and the two-systems-no-transaction problem apply to any cache in front of any store.
- DATABASE-SPECIFICChange-data-capture as an invalidation source depends on the engine: Postgres exposes logical replication slots, MySQL the binlog, and each has different guarantees about ordering and about what a schema change does to the stream. CDC also catches writes that bypass your application entirely, which application events never will.
- SIMPLIFIEDTreats one row as one key. Derived and aggregate entries — listings, counts, rendered fragments — are the genuinely hard case, and version keys are the sketch of an answer rather than a complete one.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
Securityauthorization-enforcement
Cloudmulti-region
Domains that do not exist yet
- — Distributed Systems — consistency models. "Read-your-own-writes" and "monotonic reads" are the precise names for the guarantees a cache silently removes.