Object Storage as Data Infrastructure
Buckets, keys, objects and metadata — and the four properties of that model that decide how every data pipeline above it must be written.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What does it change about a pipeline that its storage has a flat key space, no rename, per-request cost and immutable objects?
Every layer above it. Table formats need one atomic single-object write (Open Table Formats); query engines need cheap ranged reads and few round trips (The Parquet Read Path); ingestion needs to write without coordinating; and all three inherit whatever this layer does and does not promise (The Data Lake).
One object: an opaque immutable blob addressed by a key within a bucket, with a small amount of metadata attached. The object is the unit of write, of read, of cost and of atomicity — and choosing what one object contains is the highest-leverage physical decision in a lake (File Size and the Small-Files Problem).
Treat it as a file system with a network in front of it. Directories, files, renames, appends — the API looks close enough, and most code written against it works on the first day.
The publish-by-rename pattern, moving _tmp/dt=2026-08-25/ to dt=2026-08-25/ to make output appear atomically, is not atomic here: each object is copied then deleted, so readers see a partially-populated destination (Atomic Publish).
- The publish-by-rename pattern, moving
_tmp/dt=2026-08-25/todt=2026-08-25/to make output appear atomically, is not atomic here: each object is copied then deleted, so readers see a partially-populated destination (Atomic Publish). - A job appends to an existing object to add today's rows. There is no append — the object is replaced wholesale — so a concurrent writer's rows disappear with no error (Immutability as a Concurrency Strategy).
- A streaming writer flushes small objects continuously. Query time grows and no dashboard shows why, because the cost is in listing and per-object round trips rather than in bytes (File Compaction).
- A job writes files then immediately lists the prefix to build its own manifest. On some systems and some configurations the listing does not yet show every object it just wrote, so the manifest is short and the loss is silent (Missing Rows).
- A cleanup script deletes
dt=2026-08-25/expecting a directory delete. There are no directories — the delete is one request per object, it is not atomic across them, and it can partially fail leaving a prefix that looks like a complete but smaller partition. - A reader opens a Parquet footer, then issues a separate request per column chunk. Latency per request is not the problem; the *number* of requests is, and a layout producing thousands of them turns a fast scan into a slow one (Parquet Internals).
What is actually happening
- The model has exactly four pieces. A bucket is a namespace. A key is an opaque string identifying an object in it. An object is an immutable blob of bytes. Metadata is a small set of attributes attached to the object — content type, user tags, a version id (Object Storage).
- The key space is flat.
raw/orders/dt=2026-08-25/part-0000.parquetis one string, not a path through nested directories. Slashes are a display convention, and "listing a directory" is a paginated prefix scan over a sorted key index — which on some systems is not guaranteed to reflect writes that just completed, so any pipeline whose correctness depends on "list the prefix and process what is there" is relying on a behaviour rather than a guarantee (Direct Uploads and Signed Authorization, Reconciliation). - There is no rename, because there is nothing to rename — a key is not a pointer to bytes you can re-attach. A rename is a server-side copy plus a delete: two operations, two costs, no atomicity between them (Files, Paths and Names).
- Objects are immutable. There is no partial write and no append; a write either produces a complete new object at that key or does not. This is the property table formats build on, because a single-object write is the one atomic multi-byte operation available (Compare-and-Swap: The Primitive Everything Is Built On).
- Cost and latency are per request as well as per byte. A million tiny objects and one large object holding the same data cost very differently to read, and the difference is in the request count, not in the volume (Scan Cost).
- Durability comes from replication and erasure coding underneath, invisible to you. It is the strongest promise in the model and it is the reason a lake can be a recovery position at all (Object, Block and File).
Four properties, and the pipeline habits each one forbids
The model is small enough to state completely, which is unusual and useful. A bucket is a namespace, a key is a string, an object is immutable bytes, metadata is a few attributes. Everything else — directories, renames, appends — is something your file-system-shaped code is imagining.
The four properties below are where that imagination costs you. Each is stated with the pipeline habit it invalidates, because the properties on their own read as trivia and the habits are what actually break in production.
It is worth noticing how much of the machinery in the rest of this module exists to work around exactly these four. Table formats exist because there is no multi-object atomicity. Compaction exists because per-request cost punishes small objects. Manifests exist because listing is neither cheap nor reliably immediate. The properties are the cause and the tooling is the response (The Lakehouse).
- The last line is the positive one: immutability of a *single* object is what every transactional layer above is built on (Open Table Formats).
- The
etagandversion-idin the metadata are what make conditional writes and version restore possible, and they are the cheapest safety net in the model. - User tags travel with the object. Classification recorded there survives a move that a spreadsheet of prefixes does not (Data Classification).
bucket: acme-lake a flat namespace, not a filesystem root
key: raw/orders/dt=2026-08-25/part-0000.parquet
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ one opaque string; the slashes mean nothing
object: <bytes> immutable; no append, no partial update
metadata: content-type, etag, size, version-id, user tags
FLAT KEY SPACE "list a directory" = prefix scan over a sorted key index
-> forbids: treating listing as free or as instantaneous
NO RENAME rename = server-side COPY + DELETE (two ops, no atomicity)
-> forbids: publish-by-renaming-a-staging-directory
PER-REQUEST COST cost and latency scale with object count, not only bytes
-> forbids: one object per micro-batch with no compaction
IMMUTABLE OBJECT a write replaces the whole object, atomically, or not at all
-> forbids: append-to-file; enables: the table-format pointer swapWhy many small objects are pathological
The strongest intuition to break is that a dataset's read cost depends on its size. On object storage it depends at least as much on how that size is divided, because every object carries a fixed cost in requests, round trips and planning work regardless of how few bytes it holds.
The layout below is one day of clickstream written by a streaming job that flushes every thirty seconds, next to the same day after compaction. The bytes are identical. The query is identical. What changes is the number of objects the engine must list, open and read footers from before it can read a single value.
The effect compounds with columnar formats, because a reader touches each file more than once — footer, then one ranged read per column chunk it needs. A layout that multiplies files multiplies that whole pattern, which is why the small-file problem shows up as planning-and-open time rather than as transfer time (The Parquet Read Path).
- dt=2026-08-24/ (compacted, 12 files)about 86M · 12 files · skipped
- dt=2026-08-25/hour=00..23 (streamed, 120 files/hour)about 84M · 2880 files · read
- dt=2026-08-25/ (same data after compaction)about 84M · 14 files · read
- dt=2026-08-26/ (in progress)partial · 41 files · skipped
The two dt=2026-08-25 rows are the same data. Nothing about the query, the format or the volume changed — only the object count, which is a physical decision made by the writer and paid for by every reader (File Compaction).
The job writes its output under a staging prefix, then "renames" the prefix to the final location so that consumers see the partition appear all at once. On a file system this is close to atomic. Here it is a server-side copy of every object followed by a delete of every original — so the destination fills up gradually, and any reader listing during that window sees a real prefix with a fraction of the data.
The job writes its output objects directly under final, unique keys where no reader is looking for them, because readers resolve the table through a manifest rather than by listing. When every object is durable, the job commits a new snapshot naming them — one small object write, atomic by definition.
Object storage gives atomicity for exactly one thing: a single object write. Any publish scheme that needs N objects to appear together must reduce itself to that one thing, or it does not have atomicity at all. Rename-based publishing does not reduce to it, which is why it silently produces readable partial states rather than failing (Atomic Publish).
What it costs, and which lever moves which driver
Object storage cost is the one place in a data platform where the intuitive model is actively misleading. Bytes are the cheapest driver and the one everyone watches; requests are the one that surprises people, and they are driven by a decision — object size — that nobody records anywhere.
The second surprise is versioning. Enabling it on a raw prefix is good practice and it silently retains every overwritten version forever unless a separate expiry rule is configured for non-current versions. That is two settings, and teams reliably configure the first.
The lever list is short and it is the same list every time: fewer, larger objects; a prefix scheme that matches the predicate; a manifest instead of a listing; explicit lifecycle including non-current versions; and keeping readers in the same region as the bytes (What Actually Drives Data Platform Cost).
Scales with object count and with how many ranged reads each file needs. This is the driver a compaction job moves, and it moves without changing a byte of data.
Cheap per byte, permanent by default, and doubled quietly by versioning without a non-current expiry rule.
Zero until a second region or a second consumer platform appears, then immediately significant. Co-locating readers with the bytes is the only real lever (Egress: Moving Data Costs Money, Not Just Storing It).
Pure waste: every byte is read and written again to achieve something a pointer swap does for one small write.
Lifecycle rules that move data to colder classes have their own accounting, and objects moved and then read repeatedly can cost more than leaving them where they were (Storage Lifecycle).
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative and unitless. The ordering is the teaching: the top driver is object count, which is a layout decision, and the second is a checkbox nobody revisits. Neither is data volume, which is what most people watch.
Storage classes, minimum retention periods for colder classes, request accounting, per-prefix request-rate behaviour and listing consistency all differ by provider and have changed over the years. Treat the drivers above as stable and every specific rule as something to verify in current documentation for the provider and region you are actually using.
How to build it
Most important first.
- Publish by writing a small pointer object, never by renaming a directory. This is the single most important adaptation: put the atomicity where the system has it (Open Table Formats).
- Target object sizes that make a read economical rather than sizes that mirror your write cadence. Buffer and batch on the write side; compact on a schedule when you cannot (File Size and the Small-Files Problem).
- Never let a listing be your source of truth about what exists. Keep a manifest and treat the listing as an audit tool (Open Table Formats).
- Design key prefixes for the predicate consumers will actually use — usually a date — and remember the prefix is an interface that readers will hardcode (Partitioning).
- Write to a unique key per attempt, so a retry never races with the write it is retrying. Idempotency here is achieved by key choice, not by locking (Idempotent Data Pipelines).
- Set lifecycle rules deliberately per prefix, checking what depends on the data first — a lifecycle rule is a scheduled delete of something you may need — and pair versioning on raw prefixes with an expiry rule for non-current versions, because versioning protects against an accidental overwrite and not against a writer that has been wrong for a month (Storage Lifecycle, Data Retention, Backup Strategy).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Durability of a written object, which is the strongest guarantee available anywhere in a data platform.
- Atomicity of a single object write: readers see the whole new object or the previous one, never a partial body.
- No atomicity across objects. N writes are N events, and any subset can be observed (Atomic Publish).
- No ordering guarantee between concurrent writes to different keys, and no cross-key transaction of any kind.
- Listing consistency after a write varies by provider and configuration; treating it as immediate everywhere is an assumption about a product, not a property of object storage (Data Engineering and Cloud Infrastructure).
- No guarantee that the object contains what its key implies. Keys are opaque strings and the naming convention is enforced by nothing (The Data Catalog).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The layer-specific check is a manifest-versus-listing audit: everything the manifest claims exists, and everything present is either claimed or older than the orphan threshold. Run it on a schedule, not during an incident.
- It misses object *content* entirely — a present, correctly-named, correctly-sized object full of nulls passes (Data Tests).
- It also misses the window in which a listing was stale, because by audit time the listing has caught up. The evidence of that failure class is gone by the time you look for it, which is exactly why the manifest must exist in the first place.
- A completed write is readable essentially immediately at its key, so the store adds almost no freshness cost of its own.
- What it does add is a floor on *discoverability*: if consumers find data by listing, freshness is bounded by listing behaviour rather than by the write. Manifest-based discovery removes that dependency entirely.
- Frequent small writes make data available sooner and make every subsequent read slower. That exchange is the freshness cost specific to this layer and it is paid by readers rather than by the writer (Cost vs Freshness).
- Object storage has no schema, so schema evolution is entirely a property of the file format and the table format above it (Schema Evolution).
- The key layout, however, evolves and it is a breaking interface change when it does. Readers hardcode prefixes, so changing
dt=toingest_date=breaks consumers as surely as a column rename (Breaking Schema Changes). - Object metadata and tags are a place to record classification and lineage that survives the file being moved, and almost nobody uses them for that. It is cheap and it is one of the few things that stays attached to the bytes (Metadata: Technical, Operational and Business).
- Versioning lets you restore an overwritten or deleted object, which covers the accident case well and the systematic-error case not at all.
- Cross-region or cross-account replication is the defence against a bucket-level loss or a policy mistake, and it is a different mechanism from versioning — one protects an object, the other protects the bucket (Disaster Recovery).
- Recovery of a *dataset* is not recovery of objects. Restoring the objects of a half-written partition restores a half-written partition, which is why the manifest is part of the recovery unit (Planning a Backfill).
What can go wrong
- Publish-by-rename assumed atomic, producing readable partial states on every publish.
- Small-object accumulation degrading every reader, with no error and no obvious cause.
- A listing that lags a write, so a self-built manifest is short and the loss is silent.
- Partial deletes leaving a prefix that looks like a complete but smaller period.
- A lifecycle rule expiring objects a live table snapshot still references (Storage Lifecycle).
- The mitigation failing: object versioning enabled to protect raw data, quietly multiplying retained bytes because no expiry policy was set on the non-current versions (What Actually Drives Data Platform Cost).
- "It is a file system with an API." The four properties above are the difference, and every one of them has broken a pipeline written on that assumption (File Systems: From Path to Blocks).
- "Object storage is slow." Per-request latency is higher than a local disk and throughput is enormous and parallel. Workloads that read few large objects in parallel do very well; workloads that read many small objects serially do badly, and both get called "object storage performance" (Latency and Bandwidth Are Different Resources).
- "Storage is cheap, so keep everything." Bytes are cheap and requests are not, and retained bytes accumulate against a retention obligation as well as a bill (Data Retention).
- "Eventual consistency means data can be lost." A completed write is durable. What can lag is a *listing*, which is a discovery problem and is solved by not discovering data through listings.
- "Partitioning is about directories." There are no directories. Partitioning is a key naming convention that lets a reader skip prefixes, and its effectiveness is entirely about whether the predicate matches the scheme (Partition Pruning).
- Bucket and prefix policies are far coarser than the row- and column-level control the source systems had, so a copy into object storage is usually a widening of access unless something above it narrows it again (Data Access Control).
- Encryption at rest is standard and answers a narrow question; key management and who can read the objects answers the one that matters (Key Management and Encryption at Rest).
- Immutability makes deletion obligations genuinely hard: the subject's rows live inside objects that also hold other subjects' rows, so satisfying a request means rewriting objects — which is the operation with no atomicity (Deletion Requests).
Operating it
- Object count per prefix alongside bytes per prefix. Divergence between the two curves is the small-file problem, visible months before it becomes a complaint (File Size and the Small-Files Problem).
- Requests per query, split into list, head and get. A query issuing very many gets relative to its bytes read has a layout problem, not a compute problem.
- Error and throttle rates per prefix, which reveal hot prefixes where a key scheme concentrates load (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Non-current version bytes as a separate line from current bytes, so versioning does not silently double your retained storage.
- At 10x objects, listing-based planning starts to dominate query time while reading does not change at all — the first symptom is planning latency, not scan latency.
- At 100x, key-prefix design matters for request distribution as well as for pruning, because a scheme that concentrates hot keys concentrates load (Partition Cardinality).
- Bytes scale almost freely; this is the layer where volume genuinely is somebody else's problem, which is exactly why the interesting limits are all about object count and request rate instead.
- Per-request charges dominate for small-object workloads. This is the counter-intuitive one: cost tracks file count rather than data volume, and a compaction job can change the shape of the bill without changing a byte of data (File Compaction).
- Retained bytes, including every non-current version and every snapshot a table format is holding for you (Storage Lifecycle).
- Egress and cross-region reads, which appear the moment a second engine or a second region starts reading the same bucket (Egress: Moving Data Costs Money, Not Just Storing It).
- Listing calls at planning time on very large prefixes, which is one of the costs a manifest-based table format removes outright.
- You get effectively unlimited, extremely durable, cheap-per-byte storage, and you give up every file-system affordance your code was written against — rename, append, directory operations, and cross-file atomicity.
- Large objects make reads efficient and make small corrections expensive, because a correction rewrites the whole object. Small objects invert both (File Size and the Small-Files Problem).
- Versioning buys protection against accidents and costs retained bytes indefinitely unless you also configure expiry for non-current versions, which people enable the first without the second.
Object store explorer
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
s3://lake/events/dt=2026-08-25/country=DE/part-00007.parquet
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
one flat key. the slashes are characters, not folders.| Directories | |
|---|---|
| File system | Real. A directory is a node, and listing it is cheap regardless of what else exists. |
| Object store | Not real. Keys are flat strings and / is a convention, so listing a "directory" is a paginated scan of every key with that prefix. |
| So | Partitioning finely makes listing the dominant cost of a query, long before reading does. |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALFlat key space, immutable objects, no rename, and per-request cost are properties of the object-storage model itself and hold across providers and across on-premises implementations of the same API.
- CLOUD-SPECIFICListing consistency after a write, throttling behaviour per prefix, and how a server-side copy is billed differ by provider and have changed over time; a pipeline that depends on a listing reflecting its own writes immediately is depending on a provider behaviour, not on the model.
- FORMAT-SPECIFICHow badly small objects hurt depends on the file format: a columnar format issues several ranged reads per file for footer and column chunks, so its per-file overhead is higher than a line-delimited format that is read once end to end.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a store replicated across zones can offer strong durability and still lag on a listing, and what a client should conclude from a request it cannot confirm.
- — DevOps / Production Engineering owns bucket policy, lifecycle configuration and replication as infrastructure code, including the review that stops a lifecycle rule deleting a referenced object.