The question this answers
When several machines must see the same files at the same time, what does a shared filesystem actually give you and what does it charge?
Four render workers run a third-party binary that takes an input directory and writes an output directory. It cannot be taught to speak an object-storage API, and the render farm must scale horizontally.
POSIX filesystem semantics — paths, directories, byte-range reads and writes, permissions, locks — visible identically from every machine that mounts it, with the storage surviving all of them.
The same filesystem, seen from several machines
File storage is a filesystem that lives on the other end of a network protocol. The client mounts it at a path; from that moment open, read, stat, rename and flock behave the way the application already expects, except that each one is a request to a server. Nothing above the mount point has to change, which is precisely the property you are buying.
That property is worth a lot in exactly one situation: existing software that requires a path. Legacy applications, scientific and media tooling, CMS installations with a plugin directory, anything with a --input-dir flag. Rewriting them to call an object API is a project; mounting a share is an afternoon. Choosing file storage for that reason is a good decision, and it should be recorded as that reason.
It is a poor decision when it is chosen because "two services need the same data" and nobody wanted to think further. Shared mutable state across machines is a distributed-systems problem whether or not it wears a filesystem interface. The filesystem does not make it easier; it makes it *look* easier while the coordination problem is still entirely yours.
What you pay: per-operation latency and a shared failure domain
A local filesystem answers most metadata operations from the page cache in microseconds. A network filesystem answers them in a network round trip. Individually that is fine. In aggregate it is the entire performance story, because ordinary software performs a startling number of metadata operations — a build tool stats thousands of files, a directory listing of 200,000 entries is 200,000 answers, and a script that touches a file per iteration has quietly become a network-bound program.
The rule of thumb that survives every provider: file storage is fine for throughput and expensive for metadata. Streaming a 4 GB asset is efficient. Walking a deep tree of small files is not, and no throughput tier fixes it.
The second cost is blast radius. A local disk failure kills one machine. A stalled network mount hangs *every* process on *every* machine that touches the path — and it hangs them in uninterruptible I/O, so they cannot be killed and the instance cannot cleanly shut down. A fleet of ten workers becomes ten unresponsive boxes at once, and the health check, if it does not touch the mount, keeps saying they are fine.
Consistency is the third. POSIX semantics over a network are approximated, not guaranteed, and client-side caching means one machine may not see another's write immediately. Advisory locks work, but they are advisory, and their behaviour across clients and after a client crash is protocol-specific in ways that matter.
| Dimension | What you get | What it costs |
|---|---|---|
| Application changes | None — it is a path | You inherit the application's filesystem assumptions, including bad ones |
| Sharing | Many machines, read-write, concurrently | Every one of them shares a failure domain |
| Throughput | Good, and usually elastic or provisioned | Provisioned throughput is a fixed cost you pay whether you use it or not |
| Metadata operations | Correct POSIX semantics | One network round trip each — this is where the time goes |
| Consistency | Close-to-open consistency on most protocols | Client caches mean "close enough", not "the same instant" |
| Locking | Advisory locks across machines | Crash-recovery and cross-client semantics are protocol-specific |
| Failure | The provider keeps the data durable | A stalled mount hangs every process on every client, uninterruptibly |
| Price per GB | Elastic capacity, no resizing | Typically the most expensive of the three contracts |
The anti-pattern: a shared filesystem used as a queue
It appears in every organization eventually, and it always starts reasonably. Producers drop job files into /shared/incoming. Workers list the directory, pick a file, rename it into /shared/processing, do the work, move it to /shared/done. No broker to run, no new component on the diagram, and it works perfectly in a test with one worker.
What breaks is not throughput. It is that a directory listing is not a queue read. Two workers list the directory in the same second and both see the same file; the rename is atomic on a single filesystem, which is the one thing that saves this design from total collapse, but everything around it is unprotected — a worker that crashes after renaming leaves the job stranded in processing with nothing to time it out, retries need a counter that lives in a filename, ordering is whatever the listing returns, and "how deep is the backlog" is ls | wc -l over a directory that has grown to 400,000 entries and now takes a minute to enumerate.
A message queue exists because these are hard problems, and it solves them with visibility timeouts, acknowledgements, redelivery, dead-letter destinations and an actual depth metric. The Architecture domain teaches that in Message Queues. The infrastructure lesson here is narrower: when you find yourself implementing at-least-once delivery on top of rename, the filesystem was the wrong contract, and it was chosen because it did not require adding a box to the diagram.
# producer cp job-8123.json /shared/incoming/ # worker loop for f in /shared/incoming/*.json; do mv "$f" /shared/processing/ # racy: two workers may both have listed it process "$f" || true # crash here and the job is stranded forever mv /shared/processing/$(basename "$f") /shared/done/ done # backlog depth: ls /shared/incoming | wc -l (O(n) over 400k entries)
# producer
put_object s3-like://jobs/raw/8123.json
enqueue jobs-queue '{"key":"raw/8123.json"}'
# worker loop
receive jobs-queue --visibility-timeout 300 \
| process_and_ack # unacked work is redelivered automatically
# backlog depth: the queue publishes it as a metric, in O(1)The filesystem version reimplements delivery guarantees with mv and hope. The second separates the payload (object storage) from the coordination (a queue that already has visibility timeouts, redelivery and a depth metric).
Key points
- File storage buys POSIX semantics shared across machines with zero application changes — that is its whole value proposition.
- The honest reason to choose it is usually "software we cannot change requires a path", and that is a legitimate reason worth writing down.
- Throughput is fine; metadata operations are the expensive part, because each one is a network round trip.
- A stalled mount hangs every process on every client uninterruptibly — the blast radius is the entire mount group.
- Using a shared directory as a work queue reimplements delivery semantics with
rename. Use a queue for coordination and object storage for payloads.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The provider runs a filesystem and exports it over a network protocol at an endpoint inside your virtual network.
- • Each client mounts the export at a local path; the kernel's network filesystem client translates syscalls into protocol operations.
- • Reads and writes may be cached client-side according to the protocol's consistency model, which is why two clients can briefly disagree.
- • Locks are coordinated by the server, so
flockworks across machines — with protocol-specific behaviour when a client dies holding one. - • Capacity usually grows automatically; throughput is either provisioned explicitly or scales with stored size, depending on the offering.
- • Own mount options and, above all, timeouts. A hard mount with no timeout is what turns a storage blip into unkillable processes.
- • Own the security group or firewall rule on the mount port. Anything that can reach the endpoint and satisfy the file permissions can read the data.
- • Own directory sizes. Nothing warns you that a directory has reached 400,000 entries; you find out when a listing takes a minute.
- • Own backup separately — provider snapshots of a share are not the same thing as an application-consistent backup.
- • Own the decision about which zones mount it, because cross-zone mounts add latency to every single operation.
- • A stalled mount: processes block in uninterruptible I/O on every client, the instances cannot be killed cleanly, and health checks that avoid the path stay green.
- • Metadata storms — a build or a directory walk saturates operations per second while throughput graphs look idle.
- • Silent divergence between clients under caching, so machine A does not see machine B's write for a window that the application never accounted for.
- • A stranded lock held by a client that died, blocking every other worker until it expires or is cleared.
- • A permissions model mismatch: UIDs differ between containers and the share, so files written by one worker are unreadable by the next.
- • Capacity typically scales without intervention; throughput is the provisioned dimension and it is the one you must plan.
- • Metadata operations per second saturate first for small-file workloads, and adding clients makes it worse, not better.
- • Adding clients also multiplies cache-coherence traffic, so the tenth worker costs more than the second.
- • Beyond a point the answer is not a bigger share: it is to stop sharing a filesystem and give each worker its own input, fetched from object storage.
- • Access control is network reachability plus POSIX permissions, which is a weaker and coarser model than per-object identity policy.
- • Restrict the mount endpoint with a security group so only the intended workload can reach it — this is the primary control.
- • Every client that mounts it read-write can damage every other client's data; the mount group is the blast radius, so keep it small and intentional.
- • Encryption in transit is not always the default on file protocols. Verify rather than assume — see Encryption at Rest vs in Transit.
- • A share mounted into a container that runs untrusted code hands that code the whole filesystem. Isolation belongs at the mount boundary, not in the application.
- • Usually the highest price per gigabyte of the three contracts.
- • Provisioned throughput is a fixed monthly commitment on some offerings and a per-use meter on others — the shape differs and it changes the design.
- • Capacity grows elastically, which is convenient and means nothing ever prompts you to delete anything.
- • Cross-zone traffic to the share can appear as a separate data-transfer line item, which is easy to miss because it feels like local disk I/O.
- • Metadata operations per second and client-side operation latency, split per mount — the pair that explains "the share is slow".
- • Throughput against the provisioned limit, and any burst-credit balance the offering exposes.
- • Client mount state and I/O wait on each machine, because a stalled mount shows up as I/O wait long before anything reports an error.
- • The signal that lies: the storage service's own availability metric, which stays healthy while a client-side mount is wedged.
- • Object storage plus a small download step in each worker. If the application can be given a fetch step, this is cheaper, faster to scale and removes the shared failure domain.
- • A message queue for coordination and object storage for payloads — the correct decomposition whenever the share was going to be used as a work queue.
- • Bake the shared assets into the container image when they change on a release cadence rather than continuously. No mount, no share, no consistency question.
- • A database, when what is really being shared is small structured state that the team modelled as files because files were easy.
- • One machine with a local disk. If the sharing requirement was aspirational rather than real, a single worker with a fast local volume is dramatically simpler.
- • Buys zero application change; costs per-operation network latency on every filesystem call, forever.
- • Buys concurrent access from many machines; costs a shared failure domain in which one stall hangs the whole fleet.
- • Buys elastic capacity; costs the highest price per gigabyte and a provisioned throughput decision.
- • Buys familiar POSIX semantics; costs the honesty of admitting that cross-machine coordination is still an unsolved problem in your design.
What people believe, and what is true
A shared filesystem is the simplest way for two services to share data.
It is the way that requires no code change. It also couples the two services into one failure domain and leaves every coordination problem unsolved.
It behaves like a local disk.
It behaves like a local disk until you count operations. Every stat is a network call, and ordinary software makes thousands of them.
If the share is slow, buy more throughput.
Small-file workloads saturate metadata operations, not bandwidth. More throughput changes nothing; fewer, larger files does.