Making an Existing Service Stateless
The inventory, the migration order and the verification — turning a service that works on one instance into one where any instance can serve any request.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
I have a service that assumes it is the only instance. What is the actual sequence of changes that makes it safe to run several?
The service is at its ceiling on one machine and must run on three. Behaviour must not change, and the migration must be doable one piece at a time in production.
Set the replica count to three and see what breaks. Anything that breaks will be obvious, and we can fix it as it surfaces.
The failures are intermittent rather than obvious: one request in three behaves differently, which reads as flakiness rather than as a design fault.
- The failures are intermittent rather than obvious: one request in three behaves differently, which reads as flakiness rather than as a design fault.
- The scheduled nightly job now runs three times, and the effect (three emails, three invoices, three exports) is only discovered by a customer (Scheduled Jobs).
- The in-memory rate limiter now permits three times the intended rate, which nobody notices until an abusive client does (Rate Limiting).
- Users are logged out at random as their requests land on instances that never saw their session (Where Sessions Live).
- A half-finished upload sits on the local disk of an instance that is no longer receiving that user's traffic (File Uploads Through the Backend).
- The team discovers the problem list piecemeal over three weeks, in production, with users as the test suite.
What is actually happening
- Stateless Services gives the test — *if this instance dies right now, does anything become wrong, not merely slow?* — and the categories. This lesson is the migration: applying that test as an inventory and moving each item deliberately.
- Instance-local state hides in five places, and the list is remarkably consistent across services: sessions, caches, counters and limits, locks, and timers/schedulers. A sixth, local files, appears in any service that accepts uploads.
- Each of the five has a standard destination: a shared session store or a self-describing token, a shared cache (or an accepted per-instance cache where staleness is tolerable), an atomic counter in a shared store, a lock in a store that offers atomicity across processes, and a single scheduler or a leader-elected one.
- Moving state out is not free: each move converts a memory access into a network call to a dependency that can be slow, unavailable, or a new bottleneck. The migration replaces one failure mode with another, better-understood one.
- Some state genuinely cannot be moved: an open WebSocket, an in-progress multi-part upload, an in-flight long computation. These are handled by making them recoverable — reconnect, resume, re-run — rather than by making them shared (Idempotency in Backends).
- The order matters. Sessions first (they break most visibly), then schedulers and locks (they break most expensively), then counters, then caches (they break least severely, as staleness rather than as errors).
The inventory: six places state hides
The migration begins as a search, not as a refactor. Almost every instance-local dependency falls into one of six categories, and each has a conventional destination. Writing the inventory down first turns an open-ended "make it stateless" into a finite list with an order.
The last column is the one to argue about. Not every item must move — a local cache of immutable reference data is faster where it is, and moving it buys nothing but a network hop.
| Where it hides | How to find it | If it stays local | Destination |
|---|---|---|---|
| Sessions | Session middleware configured with a memory store | Users randomly logged out | Shared session store, or self-describing tokens (Token Authentication and the Revocation Problem). |
| Caches | Module-level Map/dict, memoisation decorators | Instances disagree until each entry expires | Shared cache if invalidation matters; local is fine for immutable data. |
| Counters and limits | Module-level numbers, in-process rate limiters | Limit multiplies by instance count; resets on deploy | Atomic increment in a shared store (Rate Limiting). |
| Locks | mutex, Lock(), synchronized, in-process semaphores | Protects nothing across processes | Database row lock, conditional update, or a lease with expiry. |
| Timers and schedulers | setInterval, @scheduled, background loops | Every job runs once per instance | One scheduler, platform cron, or leader election. |
| Local files | Writes outside a temp path; upload buffers | Data lost on restart or reschedule | Object storage, streamed rather than buffered (Object Storage). |
Order the migration by how badly each one fails
The five categories do not fail equally. Sessions fail visibly and immediately; schedulers fail invisibly and expensively; caches fail as staleness, which is often survivable. Sequencing by severity means the riskiest items are done first and the service is safe to scale before the work is finished.
The pipeline below is the order that most teams end up at after doing it in a worse order once. Each step is independently shippable, which matters: this migration cannot be one pull request.
- 10. Inventory
Enumerate every item in the six categories; apply the kill test to each.
fails by Skipping straight to fixes and missing the scheduler, which is the expensive one.
- 21. Sessions
Move to a shared store or tokens; verify a request served by any instance works.
fails by Leaving a local fallback path that silently activates on store failure.
- 32. Schedulers
Exactly one owner for timed work.
fails by Leader election without a lease expiry: a crashed leader means no leader.
- 43. Locks
Replace in-process locks with atomic shared operations.
fails by A distributed lock used as if it were mutual exclusion with no expiry semantics (Pessimistic Locking).
- 54. Counters and limits
Atomic increment in a shared store.
fails by Read-modify-write over the network, which is the same race with more latency.
- 65. Local files
Stream uploads to object storage; treat disk as scratch.
fails by Buffering the whole upload in the process instead, trading a disk problem for a memory one (File Uploads Through the Backend).
- 76. Caches
Decide per cache: shared, or local-and-tolerably-stale.
fails by Moving every cache to the network and adding a round trip to every request (Local vs Distributed Cache).
- 87. Verify
Two instances everywhere; kill one under load and watch.
fails by Declaring victory on the basis that nothing obviously broke.
Verification: prove it rather than believe it
hey and Docker); the method — steady traffic on stateful paths, ungraceful kill, assert correctness rather than speed — is stack-independent.The property being claimed is behaviour under instance death, and the only way to know it holds is to kill an instance while traffic is flowing. Running two replicas in development and staging is the cheapest permanent version of that test, because it turns every "works on my machine" into "works across two processes".
The test below is deliberately crude. It does not need a chaos platform; it needs one instance terminated mid-load and an assertion that nothing became wrong. Slower is fine — a cold pool and an empty cache are the expected cost. Wrong is the signal.
1# 1. Two instances minimum, everywhere. Not just production.2# docker compose up --scale api=23 4# 2. Drive steady traffic that exercises stateful paths:5# login -> read -> write -> read-back, with a real session6hey -z 60s -c 20 -H "Cookie: sid=$SID" http://localhost:8080/me &7 8# 3. Kill ONE instance, mid-load, without a graceful signal.9# SIGKILL, not SIGTERM: this tests state, not shutdown.10docker kill --signal=KILL $(docker ps -qf name=api | head -1)11 12# 4. Assertions. "Slower" passes. "Wrong" fails.13# - zero 401/403: the session survived (sessions)14# - the write is readable afterwards (no local buffer)15# - rate limit counter did not reset (shared counter)16# - the nightly job ran exactly once in the window (scheduler)17# - no orphaned files under the app's work dir (uploads)18# - latency spike, then recovery (expected: cold pool)19 20# 5. The permanent version of this test:21# assert instance count >= 2 in every environment,22# and run step 3 on a schedule.SIGKILL rather than SIGTERM is the point. A graceful shutdown test proves the drain works (Graceful Shutdown); this test proves that no correctness depends on the instance existing. They are different properties and both need testing — a service can pass one and fail the other.
How to build it
Most important first.
- Start with an inventory, not a fix. Grep for module-level mutable variables,
setInterval/timers, in-process caches, writes outside a temp path, and any lock or mutex. Write each one down with the kill test applied to it. - Move sessions to a shared store, or to tokens that carry verifiable claims. Do it first: it is the change users notice and the one that blocks everything else (Where Sessions Live).
- Give scheduled work exactly one owner — a separate scheduler process, a platform cron, or leader election among the replicas. Three replicas with three timers is three executions (Scheduled Jobs).
- Replace in-process locks with something atomic across processes: a database row lock, a conditional update, or a lease with an expiry. An in-process mutex was never a distributed lock (A Mutex on Server A Does Nothing About Server B in Concurrency & Parallelism).
- Move counters and limits to an atomic shared operation. Correctness comes from the atomicity, not merely from the sharing (Atomic Operations).
- Decide per cache whether staleness across instances is acceptable. Immutable or slowly-changing data can stay local and is faster there; anything invalidated by a write usually cannot (Local vs Distributed Cache).
- Stream uploads to object storage rather than to local disk, and treat any local file as scratch that may vanish (Presigned URLs).
- Make long-running or connection-bound work recoverable: a client that reconnects, a job that can be re-run safely, an upload that can resume.
- Verify rather than assume — see the verification section. Running two instances in every non-production environment is the cheapest permanent test there is.
What can go wrong
- Sticky sessions used as the migration instead of as an optimisation, which hides every remaining instance-local dependency until an instance dies (Sticky Sessions).
- The shared store becomes a single point of failure and a new bottleneck, and the service is now less available than it was on one instance (Cascading Failure).
- A "temporary" local file, log or lockfile that is load-bearing and is discovered only when a second instance appears.
- Leader election implemented with a lock that has no lease expiry, so a crashed leader means no leader until someone intervenes.
- Moving everything to the shared store, including data that was fine locally, and paying a network round trip on every request for state nobody shared.
- A migration that is 95% complete, where the remaining 5% is invisible and only manifests under instance failure.
- During the migration both the old local path and the new shared path can be live, so two instances can disagree about the same counter or session until the old path is removed.
- Leader election has an inherent window where the previous leader believes it is still leader; anything the leader does must tolerate a brief overlap (Backend Races).
- A counter moved to a shared store is only correct if the increment is atomic — a read-modify-write over the network is a race with extra latency (Atomic Operations).
- Session revocation must reach every instance or live in the shared store; a locally-cached session can outlive a logout elsewhere (Where Sessions Live).
- The shared session and cache store now holds data that used to be in one process's memory. It needs authentication, network restriction and encryption in transit like any other data store.
- Local files from uploads can persist on instance disks after processing, sometimes containing personal data. Ephemeral instances make this less likely and not impossible.
- A distributed lock without a lease can be held forever by a dead holder; a lease that is too short can be held by two owners at once. Both have security-relevant consequences when the lock guards a financial operation (Pessimistic Locking).
- "We moved sessions to Redis, so we are stateless." Sessions are one of six categories. Schedulers, locks and counters are the ones that cause the expensive incidents.
- "Stateless means no state anywhere." It means no durable *shared* state in the instance. Pools, warm caches and prepared statements are state, and they are the reason the process is long-lived.
- "It works with three instances in staging." Staging rarely kills instances mid-request. The property being tested is behaviour under instance *death*, not under instance *count*.
- "The load balancer will keep users on the same instance." That makes local state survive routine balancing, not deploys, scale-down or crashes (Sticky Sessions).
Operating it
- Compare the same metric across instances under identical workload. A metric that differs per instance is instance-local state, and this is the most reliable detector there is.
- Alert on any scheduled job that executes more than once per scheduled interval — a direct test for the scheduler problem.
- Log the instance identity on every line so "only on one instance" becomes a query rather than a hypothesis (Structured Logging).
- Track shared-store latency and error rate as a first-class dependency, because after the migration it is on the request path (The Metrics a Backend Must Emit).
- Once the property holds, capacity becomes a number you change — that is the entire return on the work (Horizontal vs Vertical Scaling).
- The shared stores now absorb what memory used to. They become the thing to size, watch and have a failure plan for.
- At high instance counts, per-instance overhead multiplies: each replica holds a connection pool, warms its own cache and makes its own outbound calls, so the dependencies see instance-count times the pressure (Connection Pools).
- Every move from memory to a shared store adds a network hop, a dependency and a new failure mode. In-memory was genuinely faster and genuinely simpler; correctness across instances is what you are buying.
- A service that will only ever run one instance pays these costs for nothing. Statelessness is a means to horizontal scaling and rolling deploys, not a virtue in itself.
- Recoverable-instead-of-shared designs (reconnect, resume, re-run) push complexity to the client or the job, which is often the right place and is never free.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe inventory categories are the same in every language and framework, because they follow from having a process that outlives requests.
- RUNTIME-SPECIFICPre-fork runtimes (Gunicorn workers, PHP-FPM pools) already have per-worker memory, so local state breaks at worker granularity on a single machine — the bugs appear earlier and are often misdiagnosed as flakiness rather than as an instance-count problem.
- SCALE-SPECIFICFor a genuinely single-instance internal tool that can tolerate a restart, the cost of this migration is real and the benefit is zero. Do it when a second instance, a rolling deploy or an autoscaler is actually coming.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.