The question this answers
Given how this data is written and read, which storage contract is right — and what does that choice cost me?
A new service needs somewhere to put data, and the team is about to pick whatever the last project used. The requirement is to make the decision from the access pattern instead, and to state the trade-off out loud before it is committed to.
A repeatable decision path from access pattern to storage contract, where every leaf names the complexity, the cost shape and the alternative that would have been simpler.
Four questions, asked in this order
The order matters, because the first two questions are constraints and the last two are preferences. Ask the constraints first and the answer is usually forced.
1. Does something require filesystem semantics? Not "would it be convenient" — does an application you cannot change take a path, or does an engine need fsync and in-place modification? If yes, object storage is out, and you are choosing between block and file.
2. Do several machines need to write the same data at the same time? If yes and the answer to question 1 was also yes, you are at file storage, and you should be honest that you chose it because the software required it.
3. Are the objects large, immutable and numerous? Uploads, media, backups, logs, artifacts. If yes, object storage, and the metadata index belongs in a database.
4. Is this actually structured data with queries and transactions? Then it is not a storage question at all. It is a database question, and the answer is a managed database — see Managed Databases. A surprising share of "which storage should we use" conversations end here.
Every recommendation, with its bill attached
A decision tree that stops at "use object storage" has taught nothing. The learner needs the shape of the obligation they just accepted: what they now operate, what meter starts running, and which simpler option they walked past.
Read the row for your answer, and then read the "simpler option" column again. In a majority of real cases the simpler option is adequate, and the reason it was rejected is worth stating explicitly rather than assuming.
| Choice | Because | You now own | Cost shape | Simpler option, and when it wins |
|---|---|---|---|---|
| Object storage | Large immutable blobs, read by many, written once | Key schema, lifecycle rules, versioning, reconciliation with the database | Storage + requests + egress; requests dominate for small objects | A database column, when the blobs are small and always read with their row |
| Block volume | An engine needs random writes and a durable flush | Capacity, free-space alerting, performance tier, snapshots and restore tests | Provisioned capacity and IOPS — fixed, paid whether used or not | A managed database, which owns the volume for you and usually should |
| Shared filesystem | Software that cannot be changed requires a mounted path from several machines | Mount options and timeouts, directory sizes, throughput provisioning, a shared failure domain | Highest per-GB, plus provisioned throughput | Object storage plus a fetch step in each worker, whenever the app can be given one |
| Managed database | The data is structured, queried and transactional | Schema, queries, indexes, access control, instance size and the bill | Instance-hours + storage + IOPS + backups; scales with the instance, not the traffic | Object storage plus an index table, when there are no queries beyond "get by id" |
| Instance-local disk | The data is genuinely disposable — cache, scratch, spill | Nothing. Losing it on restart is the contract. | Included in the instance price | Nothing at all: recompute the value instead of storing it |
Three worked calls, including one where the cheapest option wins
The tree is only useful against real inputs, so here are three, with the trade-off stated the way it should appear in a design document.
A photo-sharing app storing user uploads. Large, immutable, read by many, served publicly. Object storage, with a CDN in front and metadata rows in the database. The trade-off: two writes that cannot be made atomic, so a reconciliation job is now part of the system, and the request meter matters more than the storage meter.
A PostgreSQL instance for the same app. Random writes, in-place page modification, fsync on commit. Block storage — but the better answer at this size is a managed database that owns the block storage on your behalf. The trade-off: less control over the exact volume and tier, in exchange for not owning patching, failover and snapshot scheduling. See Managed vs Self-Hosted.
A build farm's compiler cache. Regenerable, hot, latency-sensitive, worthless after an hour. Instance-local ephemeral disk. The trade-off: none worth mentioning — losing it costs a slower build. This is the case where the boring answer is right, and it is the one teams most often over-engineer, usually into a shared filesystem that then becomes the slowest component in the pipeline.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Ask the constraints first — filesystem semantics, then concurrent writers — because they force the answer. Preferences come after.
- Large, immutable, numerous, read by many: object storage, with the index in a database.
- Random writes and a durable flush: block storage, and at most sizes that means a managed database that owns the volume.
- Several machines, one path, software you cannot change: shared filesystem, and say that the software forced it.
- Disposable data belongs on the instance-local disk, and some data does not need storing at all — recompute it.
- Every leaf must state the trade-off. "Use object storage" without "no transactions, and requests are a meter" is not advice.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Characterize the access pattern: unit of read, unit of write, mutability, concurrency, size distribution and latency requirement.
- • Eliminate contracts that cannot satisfy the constraints — no
fsyncon object storage, no multi-writer on a block volume. - • Among the survivors, choose the cheapest operationally, not the cheapest per gigabyte.
- • Write down the trade-off and the simpler option you rejected, so the next engineer inherits the reasoning rather than the conclusion.
- • Revisit when the access pattern changes — the decision was made against a shape, and shapes change.
- • Own the record of *why*. A storage choice with no stated reason gets copied into the next three services.
- • Own the review trigger: object counts, volume free space and share throughput are the three numbers that say the shape has changed.
- • Own migration cost awareness — moving between contracts means rewriting application code, not copying bytes.
- • Own the "do we still need this data" question, which no storage service will ever ask you.
- • Choosing by familiarity: the last project used a shared filesystem, so this one does too, and inherits a failure domain it never needed.
- • Choosing by price per gigabyte, then paying three times that in per-request charges.
- • Choosing object storage for data that needed transactions, and discovering it during the first partial failure.
- • Choosing a block volume for something several machines must write, and corrupting it the first time both mount it.
- • Choosing durable storage for data that was always disposable, and now backing it up forever.
- • Object storage is the only contract where growth is not your problem; the others all have a dimension you must provision.
- • The decision usually needs revisiting at the point where object count, not object size, crosses an order of magnitude.
- • A choice that was right at 10 GB can be wrong at 10 TB — most often a block volume that has hit its IOPS ceiling.
- • Migration between contracts is an application change, so the cost of a wrong choice grows with the amount of code that touches it.
- • Each contract has a different access model: identity policy for object storage, instance access for block, network reachability plus POSIX permissions for file.
- • Choosing a contract therefore chooses a security model. A share is coarse; per-object policy is fine-grained. That belongs in the decision.
- • Data classification should precede the choice: regulated data may rule out a contract because of where and how it can be encrypted or audited.
- • Whatever you pick, the default exposure question is the same one: who can read this without an application in the path?
- • Per gigabyte, the ordering is usually object < block < file — and it is the least useful comparison of the four.
- • Fixed versus usage-shaped is the more decision-relevant axis: block and file are provisioned, object is metered.
- • The meter that surprises depends on the contract: requests for object, provisioned IOPS for block, throughput for file, backup retention for managed databases.
- • The cheapest storage is the data you decided not to keep. A retention decision beats every tier decision.
- • Track the shape, not just the size: object count and average object size, volume free space and IOPS headroom, share operations per second.
- • Cost per service split by storage line item, which is what makes a mismatched contract visible. See Cost per Service and the Attribution Problem.
- • Growth rate rather than absolute size — the derivative is what tells you when the decision needs revisiting.
- • The signal that lies: total bytes stored, which grows smoothly through every one of the failure modes above.
- • Do not store it. Derived data that costs milliseconds to recompute rarely earns a durable home, a backup and a lifecycle policy.
- • Store it in the database you already run. One system to back up, one access model, transactions included — the right answer far more often than the size of this lesson suggests.
- • Keep it on the instance-local disk and accept losing it. For caches and scratch this is not a compromise, it is the correct contract.
- • Delay the decision: put it in the simplest place that works and revisit at a known threshold, rather than designing for a scale you have not reached. See No Cargo-Cult Infrastructure.
- • A structured decision costs an hour of design time and saves a rewrite; the cost is real and lands before any value is visible.
- • Optimizing the contract per data class is correct and multiplies the number of storage systems the team operates.
- • Choosing the simplest contract early may mean migrating later — usually cheaper than operating complexity you did not need yet.
- • Writing the trade-off down commits you to a reason that a future engineer can challenge, which is uncomfortable and is the point.
Which storage does this workload need?
R1 txn → A database skipped R2 posix AND shared → File storage skipped R3 posix OR raw → Block storage skipped R4 blobs → Object storage ✓ fired R5 default → Object storage would fire
What people believe, and what is true
There is a best storage type.
There is a best match for an access pattern. The same team correctly uses all four in one system, for four different data classes.
Start with the cheapest per gigabyte and upgrade later.
Migration between contracts is an application rewrite. The cheap-per-gigabyte option is expensive if the access pattern makes it pay per request.
This is an infrastructure decision.
It is an access-pattern decision with infrastructure consequences. If nobody can describe how the data is read, the decision is not ready to be made.