Stateless Services
Keeping request-serving instances free of durable state, and being precise about which state that means.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What does it mean for a backend to be stateless, given that every backend has state somewhere?
The service runs on one instance. We need to run several, behind a load balancer, without changing behaviour.
Keep using memory for sessions, caches and counters — it is fast, and the load balancer will send users to the same instance anyway.
A user's second request lands on instance B and their session is on instance A: they appear logged out at random.
- A user's second request lands on instance B and their session is on instance A: they appear logged out at random.
- An in-memory rate limiter allows N requests *per instance*, so the effective limit multiplies by instance count (Rate Limiting).
- A cached value updated on one instance stays stale on the others until each expires independently.
- An in-process lock protects nothing once a second process exists (Pessimistic Locking).
- Autoscaling and rolling deploys destroy instances routinely, taking their memory with them.
What is actually happening
- Statelessness is a claim about where durable, shared state lives, not a claim that the process has no variables.
- Process state — an open connection pool, a compiled regex cache, a prepared statement — is fine, because losing it costs performance, not correctness.
- Application state — sessions, counters, locks, uploaded bytes mid-processing — must live somewhere shared and durable, because losing it changes behaviour.
- The test: if this instance is killed right now, does anything become incorrect? Slower is acceptable. Wrong is not.
Process state versus application state
The word "stateless" causes more confusion than it removes, because every process obviously has state. The distinction that matters is what breaks when the process dies.
| State | Kind | If the instance dies |
|---|---|---|
| Connection pool | Process | Reconnects. Slower briefly. |
| Prepared statements / compiled regex | Process | Rebuilt. Slower briefly. |
| Local read cache of immutable data | Process | Refetched. Slower briefly. |
| User session | Application | User is logged out. Incorrect. |
| Rate-limit counter | Application | Limit resets, or multiplies across instances. Incorrect. |
| In-process lock | Application | Never worked across instances. Incorrect. |
| Uploaded file on local disk | Application | Data lost. Incorrect. |
| Scheduled-job timer | Application | Job skipped, or run once per instance. Incorrect. |
The kill test
There is a single question that resolves nearly every case, and it is worth applying literally: if I terminate this instance mid-traffic, does anything become wrong — not slow, wrong?
Slower is acceptable and expected; a cold pool and an empty cache are the normal cost of a deploy. Wrong means a user is logged out, a limit is bypassed, a file vanishes or a job is skipped. Everything in the "wrong" column has to move out of the instance.
const hits = new Map<string, number>()
function allow(key: string, limit: number) {
const n = (hits.get(key) ?? 0) + 1
hits.set(key, n)
return n <= limit
}// INCR returns the new value atomically;
// the first caller sets the expiry window.
async function allow(key: string, limit: number, windowSec: number) {
const n = await redis.incr(key)
if (n === 1) await redis.expire(key, windowSec)
return n <= limit
}The first counts per process, so three instances allow three times the limit — and the count resets on every deploy. The second is one counter for the whole service, and the atomicity of INCR is what makes it correct under concurrency rather than merely shared.
How to build it
Most important first.
- Move sessions to a shared store, or use tokens that carry their own verifiable claims (Where Sessions Live).
- Move caches to a shared cache when consistency across instances matters, and keep them local when it does not (Local vs Distributed Cache).
- Move locks and counters to the database or a coordination service — anything atomic and shared (Atomic Operations).
- Stream uploads to object storage rather than buffering to local disk (File Uploads Through the Backend).
- Keep process state deliberately: pools and warmed caches are why the process is long-lived at all.
What can go wrong
- Sticky sessions used to paper over instance-local state: it works until an instance dies and takes its users' sessions with it (Sticky Sessions).
- A "temporary" local file that becomes load-bearing.
- In-memory job scheduling, so scaling to three instances runs every scheduled job three times (Scheduled Jobs).
- Moving everything to a shared store, making the store the new single point of failure and the new bottleneck.
- An in-process counter incremented by concurrent requests races within the instance, and diverges across instances — two different bugs with one cause (Backend Races).
- Local session state can outlive a logout on other instances, so revocation must reach every instance or live in the shared store.
- Local files from uploads may persist on disk after processing, sometimes containing personal data.
- "Stateless means no state." It means no *durable shared* state in the instance. Pools and caches are state, and they are fine.
- "Sticky sessions make instance-local state safe." They make it survive routine load balancing, not instance death or a deploy.
- "We are stateless because we use JWTs." Sessions are one kind of state; caches, counters, locks and scheduled jobs are others.
Operating it
- Compare a metric across instances. Behaviour that differs per instance for the same workload is instance-local state.
- Watch for correctness that improves when traffic is pinned to one instance — a strong signal of hidden local state.
- Statelessness is what makes horizontal scaling routine: any instance can serve any request, so capacity is a number you change (Horizontal vs Vertical Scaling).
- The shared stores now carry the load that memory used to absorb, which makes them the thing to size and watch.
- Every move from memory to a shared store adds a network hop and a dependency. In-memory is genuinely faster and genuinely simpler — the cost is correctness across instances.
- Token-based sessions avoid the lookup and make immediate revocation hard (Token Authentication and the Revocation Problem).
- A single-instance service that will never scale out pays these costs for nothing. Statelessness is a means, not a virtue.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALApplies wherever more than one instance serves the same traffic.
- SCALE-SPECIFICIrrelevant at exactly one instance that never restarts during a request — which is no production system, but is many internal tools, and there the costs are real and the benefits zero.
- RUNTIME-SPECIFICPre-fork runtimes (Gunicorn workers, PHP-FPM) already have per-worker memory, so "local state" breaks at worker granularity inside a single machine — often surfacing the bug earlier and more confusingly than a single-process Node service would.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.