Caching in Backends
A cache trades correctness-in-time for work avoided; everything else in this module is about controlling that trade.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What does a cache actually buy a backend, and what does it cost that a faster query would not?
The product page loads slowly. It runs the same six queries for every visitor, and the answer barely changes between visitors.
Put Redis in front of the slow query. Read from Redis, fall back to the database, and the page gets fast.
The page gets fast and then someone edits a product. The old title is served for as long as the entry lives, and nobody can say how long that is without reading the code.
- The page gets fast and then someone edits a product. The old title is served for as long as the entry lives, and nobody can say how long that is without reading the code.
- Redis becomes a hard dependency you did not plan for. When it is unreachable, every request falls through to a database sized for cached traffic, and the site goes down harder than it ever did when it was merely slow.
- The hit rate is 20% because the key includes a user id and most users visit once. You now pay a network round trip on every request to avoid one query in five.
- Two services cache the same row under two different key formats. Invalidating one does not touch the other, and the bug reproduces on one page and not the other.
What is actually happening
- A cache is a second copy of data that is allowed to be wrong for a while. That is the entire abstraction. Hit rate decides how much work you avoid; staleness window decides how wrong the copy may be.
- The saving is not only the database query. It is the query planning, the row fetch, the object mapping, and often several dependent round trips collapsed into one lookup of an already-serialized value.
- A cache does not reduce load — it *moves* it. Work avoided at the database becomes work done at the cache, plus a new network hop, plus the CPU of serializing and deserializing the value.
- Every cache has four decisions baked in whether you make them consciously or not: what the key is, what the value is, how entries expire, and how entries are evicted when memory runs out. Eviction is the one people forget: under memory pressure a cache silently drops entries you expected to be there (TTL and Expiry).
- Caches exist at every layer of a request already — CPU caches, the database's buffer pool, the ORM's identity map, the HTTP layer, the CDN. Adding an application cache means choosing to add a fifth one, and it should be the layer where the work you are avoiding is actually expensive.
The cache sits on the read path, and only on the read path
The picture worth holding is narrow: a cache is a branch in the read path with a fast arm and a slow arm. All of the difficulty lives in what happens on the slow arm and what happens when the underlying data changes while the fast arm is still answering.
Notice what the diagram does *not* show. There is no arrow from the database back to the cache. The database does not know the cache exists and will never tell it that a row changed. That absent arrow is the whole of Cache Invalidation, and it is why caching is a correctness topic rather than a tuning knob.
You already have caches; pick the layer deliberately
Before adding one, it is worth being precise about which caches are already absorbing this work. A query that hits the database's buffer pool is not touching disk. A repeated fetch inside one request may already be served by the ORM's identity map. Adding an application cache in front of work that is already cheap adds a hop and a staleness window and saves nothing.
The layer you choose determines what you can invalidate. A CDN entry is invalidated by a purge API and a propagation delay you do not control; an in-process entry is invalidated by a variable assignment on one instance and nowhere else. Reach for the layer whose invalidation story matches how urgently the data must be correct.
| Layer | What it avoids | Who invalidates it | Typical staleness you accept |
|---|---|---|---|
| CPU / memory hierarchy | Main-memory round trips | Hardware, automatically | None — it is coherent (Cache Coherence: Why Shared Memory Works At All) |
| Database buffer pool | Disk reads for hot pages | The database, on write | None — it is the database's own copy |
| ORM identity map | Repeated fetches of the same row *within one request* | End of the unit of work | One request (What an ORM Actually Does) |
| In-process application cache | The whole query plus mapping, on this instance | You, on this instance only | Seconds, and per-instance divergence |
| Distributed cache (Redis/Memcached) | The whole query plus mapping, fleet-wide | You, explicitly, from any instance | Seconds to minutes, uniformly |
| HTTP / CDN | The entire request reaching your service | Cache-Control, or a purge with propagation delay | Minutes to hours (CDN as Infrastructure) |
Is this thing cacheable at all?
The decision is not "is this slow". It is whether the data has the three properties that make a cache pay: it is read far more than it is written, it is read *repeatedly* (the same keys recur), and someone can state a tolerable staleness in seconds. Fail any one and the cache is a liability.
Run this before choosing a cache technology, not after. Most of the arguments about Redis versus an in-process map dissolve once you notice the data in question is written on every read, or that the key space is one entry per user per visit.
What is true about this data?
when Product catalogue, feature flags, currency rates, config, permission sets.
cost A staleness window and an invalidation path you now maintain.
when Aggregations, report rollups, rendered fragments, search facet counts.
cost Cache memory proportional to the input space; recomputation is a spike, not a trickle (Cache Stampede).
when Account balances, inventory counts at checkout, anything a user can dispute.
cost Do not cache the value. Cache the immutable parts around it and read the number fresh (When Not to Cache).
when Per-request search strings, one-shot tokens, user-specific dashboards visited once.
cost Near-zero hit rate. You pay the lookup and the memory for nothing.
when A single indexed primary-key lookup.
cost The cache round trip is comparable to the query it replaces, and you have added a consistency problem to buy it.
How to build it
Most important first.
- Start from the query, not the technology. Measure what the expensive step is before deciding a cache is the answer (Why Is My API Slow?) — a missing index is cheaper to add and has no consistency cost (Should I Add an Index?).
- Name the staleness budget out loud, in the ticket: "a product title may be up to 60 seconds out of date". If nobody will accept a number, you have discovered that this data should not be cached (When Not to Cache).
- Choose a key format once, put it behind a function, and never build keys by string concatenation at call sites. Key drift is the single most common cause of un-invalidatable entries.
- Cache the shape you serve, not the shape you store. Caching a fully-rendered DTO avoids the mapping cost too; caching a raw row saves only the fetch (Three Models, Not One).
- Decide up front what happens when the cache is down: serve from the origin, or fail. Both are valid. Choosing by accident is not.
- Include a version or schema marker in the key so that a deploy which changes the value shape cannot read old entries as the new type (Cache Invalidation).
What can go wrong
- Cache unreachable, so every request falls through and the database sees traffic it has not been sized for since the cache was introduced.
- Memory pressure triggers eviction, hit rate collapses, and latency returns to the uncached baseline with no error anywhere (Leak or Unbounded Cache? The Question That Picks the Fix).
- A deploy changes the serialized value shape, and running instances deserialize the old shape into the new type — usually a field that is now
undefinedrather than an exception. - The cache holds a value the caller was never allowed to see, because the key omitted the tenant or the user (Tenant Isolation).
- Caching negative results without meaning to: an entry stores
nullfrom a failed lookup, and the record stays "missing" for the whole TTL after it is created.
- Two requests miss the same key simultaneously and both compute the value. Usually harmless, occasionally a thundering herd against the origin (Cache Stampede).
- A write and a read interleave: the reader loads the old row, the writer commits and invalidates, then the reader populates the cache with the value it read before the write. The cache is now stale with no TTL expiry to save it (Cache Invalidation).
- Two writers invalidate and repopulate in opposite orders, leaving the older value resident (Backend Races).
- Any per-caller data in a shared cache needs the caller's identity in the key. A key of
user:profilerather thanuser:profile:{id}serves one user's profile to everyone, and it will look like a caching bug rather than a data breach (Object-Level Authorization). - Authorization must be re-evaluated on the cached path. Caching the *result of an authorized read* and then serving it to a different caller skips the check entirely (Where the Check Belongs).
- Do not cache secrets, tokens or full credential objects. A cache is usually less protected than your database: fewer audit controls, often no encryption at rest by default, and frequently reachable from more services (Secrets Are Not Configuration).
- Cache servers are frequently deployed without authentication inside a private network. That is one network misconfiguration away from an open read of everything you have cached (Public Exposure, Read With Context).
- "Every backend needs Redis." Most backends at most scales are correct and fast without any application cache. A cache is a response to a measured cost, not a component of a reference architecture.
- "The cache made it fast, so the query is fine." The uncached query is still there, and it now runs on every miss, every eviction and every cold start — usually all at once (Cache Stampede).
- "Caching is a performance concern." It is a correctness concern with a performance benefit. The interesting bugs it causes are wrong answers, not slow ones.
- "A high hit rate means the cache is working." A high hit rate on data nobody would have re-read anyway means you built a very efficient way to avoid free work.
Operating it
- Hit rate per key prefix, not globally. One global number averages a 99% hit prefix with a 3% one and tells you nothing about either (A 95% Hit Rate Tells You Almost Nothing).
- Origin load with and without the cache: the metric that matters is queries per second reaching the database, not the cache's own throughput.
- Latency of the cache lookup itself, as its own span. A slow cache is a pure loss — you pay the lookup and still do the work (Tracing From the Backend's Side).
- Eviction count and memory usage. Rising evictions with a falling hit rate means the working set no longer fits and the TTL is now decorative.
- Key cardinality. An unbounded key space (a raw query string, a timestamp, a free-text search term) produces a cache that never hits and always fills.
- At 10x traffic on the same data, caching gets *better*: the same working set serves more requests, so hit rate rises and origin load stays roughly flat.
- At 10x *data* with the same traffic, caching gets worse: the working set stops fitting in memory, evictions rise, and you are paying for a cache that mostly misses.
- Hot keys concentrate: at high scale a small number of keys can saturate a single cache node or connection while the rest of the cluster is idle (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Above one instance, an in-process cache becomes N independent caches with N independent staleness windows (Local vs Distributed Cache).
- You accept a consistency problem you did not have. Every cached read can now be wrong, and "how wrong, for how long" becomes a property of your system that someone has to own.
- You accept an availability dependency. The cache is now on the critical path for latency, and possibly for correctness if you cache things you cannot recompute.
- You accept operational surface: memory limits, eviction policy, connection pools to a second data store, and a new thing to page on.
- Cached code is harder to reason about locally. "Why is this value stale" is a question that spans a deploy, a TTL, an invalidation path and an eviction policy.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe key/value/expiry/eviction model and the staleness trade hold for any cache, in-process or remote, in any language.
- SIMPLIFIEDTreats the cache as one layer. Real requests pass through several — CPU cache, database buffer pool, ORM identity map, HTTP cache, CDN — and the one you add may be duplicating work another already avoids (What a Cache Actually Is).
- CLOUD-SPECIFICManaged cache products differ in what they guarantee on failover: some promise persistence and replica promotion, others treat the whole dataset as disposable. Whether a restart empties your cache is a product-level fact, not a general one.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.