The question this answers
Where is a workload allowed to write, and what happens to everything else it writes?
The checkout service accepts uploaded receipts, writes a session cache, appends an audit log and stores order records. Any replica may be restarted, rescheduled onto another node or deleted during a rollout, at any moment, without warning.
A clear rule for every write the application makes — disposable, node-local, or durable elsewhere — so a routine container replacement never becomes data loss.
The writable layer exists until it does not
A running container gets one thin writable layer on top of the image's read-only layers. Writes go there, reads fall through to the layers beneath. It behaves like a normal filesystem, which is precisely the problem: nothing about writing to /app/uploads signals that the directory disappears when the container does.
And containers are removed constantly, mostly on purpose. Every rollout replaces every replica. Every scale-in deletes some. A failed health check triggers a restart. A node upgrade drains everything on it. In an orchestrated environment a container may live minutes. The container is the unit of replacement, and treating its filesystem as storage means treating a replacement — a normal, successful, uneventful operation — as data loss.
The failure is quiet, which is what makes it dangerous. No component failed. No alarm fired. The deployment was green. The receipts uploaded in the last hour are simply not there, and the first evidence is a support ticket days later.
- 1Container created
Image layers are mounted read-only; one empty writable layer is added on top.
The application assumes the directory from the last run still has files in it. It never does.
- 2Application writes
Files land in the writable layer, or in a mounted volume if the path was mounted.
Nothing distinguishes the two at the filesystem level — only the mount configuration decides which one you got.
- 3Restart in place
Some runtimes keep the writable layer across a restart of the same container object.
This is what convinces a team the data is durable. It survives the restart and not the replacement.
- 4Replaced by a rollout or scale-in
The container object is deleted and a new one created, with a fresh empty writable layer.
Everything written to the container filesystem is gone. No error, no alert, no failed component.
- 5Rescheduled onto another node
The workload starts on a different machine.
Even a node-local volume does not follow it. Local disk pins a workload to a node — that is the tradeoff, and it is real.
- 6Node lost
The machine and its local disks are gone.
Only storage in a separate failure domain — network volume, managed database, object store — survives this. See Failure Domains.
Four homes for state, and how to pick one
Every write an application makes belongs in one of four places, and the choice follows from two questions: must it survive the container, and must it survive the *node*.
The container filesystem is correct for genuinely disposable data: a scratch file during request processing, a decompression buffer, a local cache that can be rebuilt. Mount it as tmpfs or make the root filesystem read-only with an explicit writable scratch path, so the intent is visible in the manifest rather than in someone's head. A volume — a network-attached block device — survives container replacement and, if network-attached rather than node-local, can follow a workload to another node. It is the answer when the workload genuinely needs a filesystem: a database's data directory, a search index. A managed database is the answer for structured, queryable, transactional data, and it is the answer far more often than teams starting from a container mindset assume (Managed Databases). Object storage is the answer for uploads, exports, backups and anything large and blob-shaped — it is durable across zones by default, has no capacity to manage, and removes the write entirely from the compute layer (Object Storage).
The strong default for a stateless service is that it writes nothing durable at all. Logs go to stdout, uploads go to object storage, sessions go to a shared cache or a signed token, records go to the database. That is not asceticism — it is what makes replicas interchangeable, which is what makes rollouts, autoscaling and node replacement safe (Stateful Workloads: Databases Are Not Stateless APIs).
The decision, one row per kind of write
Run this table over an existing service and the mismatches are usually obvious within minutes. The three that show up most often: uploads written to a local path, logs written to a file inside the container, and an in-process session store that works perfectly until a second replica exists (Load Balancers as Infrastructure will happily send the next request elsewhere).
One honest caveat about volumes. A volume solves durability and does not solve concurrency. Two replicas mounting the same block device and writing the same files is corruption, not sharing — block storage is single-writer. If several replicas must write the same data, the answer is a database, an object store or a shared file service, never a shared block volume (Block Storage, File Storage).
| What is written | Must survive container? | Must survive node? | Correct home | What happens if it stays in the container |
|---|---|---|---|---|
| Request scratch files, decompression buffers | No | No | tmpfs or a declared scratch path | Nothing. This is the correct use. |
| Application logs | Yes | Yes | stdout → log pipeline | Lost on every replacement, and unreadable during the incident you need them for. |
| User uploads, generated exports | Yes | Yes | Object storage | Silent data loss on the next rollout; nothing fails, nothing alerts. |
| Session state | Yes | Yes | Shared cache or a signed token | Users are logged out whenever a replica is replaced or the balancer picks another one. |
| Orders, invoices, audit records | Yes | Yes | Managed database with backups and restore tests | Catastrophic and unrecoverable. This is the one that ends companies. |
| Database data directory | Yes | Yes | Network block volume with snapshots | Total data loss on node failure; local disk is a single failure domain. |
| Rebuildable local cache | No | No | Container filesystem or tmpfs | Nothing — a cold cache after replacement is a latency blip, not a loss. |
Key points
- The container filesystem is disposable by design; every rollout, scale-in and node upgrade destroys it deliberately.
- A restart may preserve the writable layer; a replacement never does — which is why teams believe the data is durable until it is not.
- Four homes for state: disposable scratch, a volume, a managed database, object storage. Pick by "must survive the container" and "must survive the node".
- Losing container-filesystem data is silent: no component fails, no alert fires, the deployment is green.
- A block volume is single-writer. Two replicas sharing one is corruption, not sharing.
- A stateless service writes nothing durable locally — that is what makes its replicas interchangeable and its rollouts safe.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The runtime mounts image layers read-only and adds one writable layer, so writes are copy-on-write against the image beneath.
- • A volume mount replaces a path in that union with storage managed outside the container, so writes to it bypass the writable layer entirely.
- • Node-local volumes live on the host's disk and therefore pin the workload to that node; network-attached volumes can be detached and reattached elsewhere.
- • Attaching a network volume to a new node takes time — detach, attach, mount, filesystem check — which is why stateful failover is slower than stateless replacement.
- • Object storage and managed databases are reached over the network with an identity, so no storage is attached to the compute at all and any replica can reach them.
- • Deleting a container removes the writable layer immediately; there is no recovery step and no recycle bin.
- • You own the audit of what your application writes and where. Grep the source for file writes — the answer is usually surprising.
- • You own making intent explicit: a read-only root filesystem with declared writable paths turns an accidental local write into a startup error instead of a silent one.
- • You own backups and restore tests for everything durable. A snapshot you have never restored is not proven recovery — see Restore Testing.
- • You own volume lifecycle: whether a volume is deleted with its workload, and whether that policy is the one you want at 3am.
- • You own the migration path when a workload outgrows a volume — usually onto a managed service, and usually with downtime.
- • Uploads disappear after a deploy. Nothing failed; the receipts were written to the container filesystem and the rollout replaced every replica.
- • Users are logged out at random because sessions live in process memory and the load balancer sent the next request to a different replica.
- • A node is drained for a kernel patch and a workload with a node-local volume cannot be scheduled anywhere, because its data is on the drained machine.
- • Two replicas mount the same block volume and the filesystem corrupts; the symptom is inconsistent reads long before anything reports an error.
- • The container disk fills with logs written to a file inside it, and the workload starts failing writes it never checked the return value of.
- • A volume is deleted along with its workload because the reclaim policy said so, and the database that used it is gone with it.
- • Stateless replicas scale horizontally without coordination — this is the entire payoff of keeping state outside.
- • A workload with a node-local volume does not scale horizontally at all: it is pinned to one node and one disk.
- • Network volumes scale in capacity but each is still single-writer, so replicas need one volume each and coordination on top.
- • Object storage and managed databases scale independently of the compute tier, which is why they are the default answer for durable state.
- • What runs out first for a stateful container workload is scheduling flexibility — long before capacity does.
- • A read-only root filesystem with explicit writable paths blocks a large class of post-compromise behaviour: no dropped binaries, no modified application files.
- • A mounted host path is a boundary hole. A container that can write to a host directory can frequently escalate to host control — see Container Security and Its Limits in the Security domain.
- • Volumes outlive containers, so a deleted workload can leave readable data behind; encrypt at rest and manage volume deletion deliberately (Key Management and Encryption at Rest).
- • Object storage access should be a scoped, prefix-limited identity, not a shared credential with full bucket access (Least Privilege in Infrastructure).
- • Data written to a container filesystem is invisible to your backup, retention and deletion processes — which is a compliance problem as well as a durability one.
- • Volumes bill for provisioned capacity whether used or not, and continue billing after the workload is gone if nothing deletes them.
- • Snapshots accumulate quietly and are one of the most common forgotten line items — see Storage Lifecycle: Hot, Warm, Archive, Delete.
- • Object storage bills for what is stored plus what is retrieved, and is dramatically cheaper per gigabyte than block storage for blob-shaped data.
- • A managed database costs more per gigabyte than either and buys backups, replication and recovery you would otherwise build and staff yourself.
- • Container disk usage on the writable layer, which finds the workload writing locally before the node fills up.
- • Volume attach and detach events and their duration — the hidden term in stateful failover time.
- • Backup success *and* restore test results. Only the second one proves recovery.
- • Object storage error rates and access-denied counts, which is what a broken workload identity looks like from the storage side.
- • The signal that lies: a successful deployment. It reports that containers started, and says nothing about what the replaced ones took with them.
- • Write nothing durable at all. For most services this is achievable and removes the entire problem — the simplest answer and usually the right one.
- • Object storage instead of a volume for anything blob-shaped: durable across zones, no capacity to provision, no single-writer constraint, no node affinity.
- • A managed database instead of running a database in a container with a volume. You are otherwise signing up to operate storage, backups, failover and upgrades yourself — see Managed vs Self-Hosted.
- • A plain VM with an attached disk when the workload is genuinely stateful, singular and does not benefit from orchestration. Not everything needs to be containerized.
- • For a batch job, writing to the container filesystem and uploading the result at the end is correct and simple. Ephemeral is a feature when the lifetime is bounded.
- • Stateless containers buy interchangeable replicas and safe rollouts; they cost a network hop to reach state that used to be local.
- • A volume buys durability across container replacement and costs scheduling flexibility, attach latency and a single-writer constraint.
- • A managed database buys backups, replication and recovery, and costs money and a boundary you no longer control — see Shared Responsibility.
- • Object storage buys durability and elasticity and costs a different access model: no filesystem semantics, no partial writes, higher per-operation latency.
What people believe, and what is true
Data written in a container persists — I restarted it and it was still there.
A restart of the same container object may keep the writable layer. A replacement never does, and replacement is what rollouts, scale-in and node upgrades do.
A volume makes a container stateful-safe.
It makes the data durable. It also pins scheduling, is single-writer, and needs backups and restore tests of its own.
Containers cannot run stateful workloads.
They can, with network volumes, stable identities and ordered rollout. The question is whether operating that yourself beats a managed service — usually it does not.
Logging to a file inside the container is fine, we collect the files.
You collect them until the container is gone, which is exactly when you needed them. stdout is the only path that survives.