The Data Lake
One place to land structured, semi-structured and unstructured data before anyone knows which questions it will answer — and the reason most lakes become swamps.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Where do you put data whose schema you do not control, at a volume a database will not hold, before you know what it will be used for?
Two groups with opposite needs. Transformation jobs want the bytes exactly as they arrived, laid out so a run can read one day without reading three years. Analysts and scientists exploring a source for the first time need to *find* the dataset, know who owns it and know what one row means — and that second need is the one a bucket serves not at all by default.
One object: a file under a key prefix holding many records. The grain that matters is not the file but what one record inside it represents, and the lake stores nothing that tells you. A prefix full of Parquet files is grainless until a catalog, a contract or a transformation declares the grain (Grain: What Does One Row Represent?).
Create a bucket, grant every team write access, tell them to land whatever they have and promise to sort it out later. This is genuinely the right first move: storage is the cheapest component in the platform, the alternative is discarding data you cannot recreate, and forcing a schema onto a source before you understand it is how you lose the field a question next quarter needs.
Eighteen months later there are four prefixes that all look like orders — raw/orders/, raw/orders_v2/, staging/orders_fixed/, exports/orders_final/ — and the only person who knew which one is authoritative has left. Nothing in the bucket records that (Source of Truth).
- Eighteen months later there are four prefixes that all look like orders —
raw/orders/,raw/orders_v2/,staging/orders_fixed/,exports/orders_final/— and the only person who knew which one is authoritative has left. Nothing in the bucket records that (Source of Truth). - A producer changes a field from an integer to a string. The files still land, the reader still parses them, and every partition written after the change has a different type for the same column. The failure surfaces months later as a query error on a range that spans the boundary (Schema Evolution).
- An hourly writer produces thousands of tiny files per day. Every query now spends most of its time listing keys and opening objects rather than reading rows, and the fix — compaction — is a job nobody scheduled because nobody owned the prefix (File Size and the Small-Files Problem).
- A job half-writes a partition and dies. The prefix now contains a mixture of complete and partial files, and a reader has no way to distinguish them, because "the set of files that make up this table" is not something a bucket records (Atomic Publish).
- A deletion request arrives for one customer. The row exists in some unknown subset of several million immutable objects, and there is no index from customer to file (Deletion Requests).
- A finance question needs last March. Somebody re-ran a transformation over that prefix in place last summer, and the original bytes are gone — so the number is not reproducible and the discrepancy cannot be explained (Keeping Raw History: The Recovery Position and the Liability).
What is actually happening
- A data lake is not a technology. It is object storage plus a convention: a bucket, a key naming scheme that encodes dataset and time, files in some format, and — if you are lucky — a catalog that maps a name to a prefix (Object Storage as Data Infrastructure).
- What makes it a lake rather than a database is that schema is applied on read, not on write. The store accepts any bytes. The meaning is supplied later by whatever reads them, which is exactly why it can hold JSON payloads, Parquet extracts, CSV drops, images and PDFs side by side (CSV, JSON and Their Limits).
- That property is the whole value proposition and the whole failure mode. Deferring the schema decision is what lets you land a source you do not yet understand. Deferring it *forever* is what produces four prefixes named orders.
- A lake stores no table concept. There is no row count, no transaction, no current version, no list of which files constitute a dataset — only keys and bytes. Everything a database gives you for free is either absent or must be rebuilt on top (Open Table Formats).
- The swamp is therefore not a discipline failure that careful teams avoid. It is the default output of the system given no additional machinery, because nothing in a bucket pushes back on an unowned, undocumented, untyped prefix (Data Platform Anti-Patterns).
A lake is a keyspace and a convention
Draw a lake honestly and it is disappointingly small: sources write objects into a bucket under keys that encode dataset and period, and readers open those objects. There is no server in the middle enforcing anything, which is precisely why it accepts everything and promises nothing.
Everything people associate with a lake — that it holds unstructured data, that it is cheap, that it scales — follows from that one design. Object storage takes any bytes, so any data type fits. It charges per byte and per request rather than per provisioned server, so idle data is cheap. It has no single node in the write path, so volume is somebody else's problem.
What also follows is the list of things that are absent. There is no table. There is no transaction. There is no notion of "the current version of this dataset". Those are not oversights — they are the price of accepting anything — and every layer above the lake exists to reintroduce one of them.
raw/is the recovery position for the whole platform: if it is intact and transformations are deterministic, mistakes are re-runs.clickstream/shows the small-file pathology forming — the writer is fine, the reader will not be (File Size and the Small-Files Problem).support_tickets_json/is schema-on-read in its purest form: it will parse until the day it does not.scratch/is how the swamp starts. It is one prefix today and forty in two years, and none of them can be deleted safely.
s3://acme-lake/
raw/ immutable, append-only, never rewritten
orders/
ingest_date=2026-08-24/ part-0000.parquet part-0001.parquet
ingest_date=2026-08-25/ part-0000.parquet
clickstream/
dt=2026-08-25/hour=13/ part-0000.parquet ... part-0119.parquet <- 120 files/hour
support_tickets_json/
dt=2026-08-25/ dump.json.gz <- schema unknown
curated/ table-format managed, atomic publish
fct_orders/
scratch/
analyst_jane_2025_q3/ no owner, no catalog entry, unknown consumersThe swamp is the default, not the exception
It is tempting to read "data swamp" as a warning about sloppy teams. It is not. A swamp is what a bucket becomes when nothing external pushes back, and nothing in a bucket pushes back. A database rejects a row that violates a constraint; object storage accepts every byte you send it, forever, from anyone with write access.
Each row below starts as a reasonable local decision — land it now and organise later, add a fixed copy rather than break the original, write hourly because the source is hourly. None of them is wrong on the day it is made. The cost lands on a different team, months later, as a question nobody can answer.
The response column is the useful part. Notice that almost every response is a piece of machinery introduced *before* the failure, not a cleanup afterwards: catalogs, owners, contracts and compaction are all cheap to add early and archaeology to add late.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A team lands a new source with no catalog entry. | A year later, four prefixes look like the same dataset and nobody can say which is authoritative. | Nothing in the write path requires registration, so registration is optional, so it does not happen. | Register at creation: a prefix without a catalog entry and a named owner does not get write access (Data Ownership). |
| A producer changes a field type without notice. | Queries spanning the change date fail, or worse, silently coerce to null on one side. | Schema-on-read means nothing validated the write, so the mismatch is stored rather than rejected. | Fingerprint the schema of each landed partition and alert on change; fail the *transformation*, not the landing (Contract Enforcement). |
| A streaming writer flushes every few seconds. | Queries get steadily slower with no change to the data volume or the query. | Planning cost scales with object count; each object also costs a request and an open. | Compact on a schedule from the day the writer is deployed, and monitor object count per prefix (File Compaction). |
| A transformation job fails halfway through writing a partition. | A day's numbers are plausible but low, and no check fires because rows exist. | Writing N objects is N independent events; nothing marks the set as complete. | Publish atomically — write to a temporary prefix and swap, or use a table format whose commit is a single pointer change (Atomic Publish, Open Table Formats). |
Someone re-runs a fix over raw/ in place. | A historical report changes and the discrepancy cannot be explained. | Raw was treated as a working area rather than as evidence. | Make raw write-once by policy, not by convention — deny overwrite at the access-policy level (Keeping Raw History: The Recovery Position and the Liability). |
| A lifecycle rule expires objects after a fixed period. | A quarterly reconciliation job finds the period it needs is gone. | Retention was set as a storage decision without asking what depends on the data. | Argue retention as a recovery-window decision per dataset, with the consumer in the room (Data Retention). |
What turns a prefix into a dataset
The difference between a lake and a swamp is not tooling spend. It is whether four questions have answers for every prefix: who owns it, what one row means, what it promises, and who reads it. A prefix with those four answers is a dataset. A prefix without them is storage cost with a future incident attached.
The comparison below is the same lake in both columns — same bucket, same files, same format. What differs is whether the answers exist and where they live. That is why "we already have a lake, we just need to organise it" understates the work: organising it is the entire product.
None of this requires a vendor. A catalog can be a table. Ownership can be a file in the repository that creates the prefix. The failure is not the absence of a platform, it is the absence of a rule that makes the answer mandatory at the point of creation, when it costs one line and not one quarter.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Every expected partition exists for the last N periods. | The producer ran when it was supposed to. | A stopped writer, a scheduler outage, a source that went away. | A partition that exists but is empty or partial; a writer that creates the key before filling it, which makes the check pass at the worst possible moment. |
| File count and byte total per partition within their own historical band. | This period looks like a normal period for this dataset. | Partial writes, a source outage, and a runaway small-file writer in the other direction. | Any error that preserves volume — wrong values, duplicated rows offsetting missing ones, last week's data re-landed under today's key. |
| Schema fingerprint of the newest partition equals the previous one. | The producer has not changed the contract. | Added, removed and retyped fields at the moment they land, rather than at the query that spans them. | Semantic change with no structural change — amount switching from gross to net fingerprints identically (Semantic Changes). |
| Row count in the landed partition equals the count the source reports for the period. | Completeness against the system of record. | Missing rows, an extract window that closed early, a paginated pull that stopped short. | Any period still open at check time; duplicates that coincidentally offset losses; and it is unavailable entirely for sources that cannot be counted (Reconciliation). |
Run these against the raw zone, not only against the serving tables. A fault caught at landing has one suspect; the same fault caught at a dashboard has twenty.
Files land under `raw/orders/`. The producing job lives in some repository. The schema is whatever the last writer emitted. A reader discovers the layout by listing keys and opening one file. When the numbers look wrong, the investigation starts by asking in a chat channel who owns this.
The same files, plus: a catalog entry naming the prefix, the owning team, the declared grain and the partition scheme; a recorded schema fingerprint per partition; a completeness and freshness check that runs on a schedule; and a table format that publishes atomically so a reader never sees a half-written state.
Every one of those additions answers a question that is otherwise answered by a person's memory. Memory does not survive re-orgs, and the failures in this domain are discovered months after the decision that caused them — so the only durable form of the answer is one that is stored next to the data and checked automatically (Dataset Documentation).
How to build it
Most important first.
- Make the raw zone immutable and append-only. Writers add partitions; nothing rewrites them. This is the property that turns a transformation bug from a data loss into a re-run (The Raw Landing Zone, Keeping Raw History: The Recovery Position and the Liability).
- Give every prefix a named owning team and a classification before the first byte lands. An unowned dataset has nobody to answer "is this still produced?", and an unclassified one will be assumed non-sensitive — an assumption that is wrong often enough to be dangerous (Data Ownership, Data Classification).
- Register datasets in a catalog at creation time, not as a later cleanup project. Retro-cataloguing a lake is archaeology, and it is the project that never gets funded (The Data Catalog, Data Discovery).
- Encode the partition scheme in the key layout and treat it as an interface, because readers will hardcode it. Date-partition almost everything; that is the predicate nearly every analytical query carries (Partitioning, Partition Pruning).
- Prefer a self-describing binary format over text for anything queried more than once. Types travel with the data, columns can be skipped, and the "is this column a string this month?" class of bug largely disappears (Parquet).
- Put a table format over any prefix that consumers query directly. The moment a dataset needs atomic publish, deletes or concurrent writers, it needs a manifest, and bolting one on later means rewriting every reader (The Lakehouse).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Durability of individual objects, which object storage takes seriously and is genuinely strong. That is the only guarantee a bare lake offers (Object Storage).
- No atomicity across objects. Writing four files is four independent events, and a reader can observe any prefix of them (Atomic Publish).
- No schema guarantee whatsoever — not across files, not across partitions, not within a file unless the format itself carries types (Parquet Internals).
- No completeness guarantee. "Everything the source produced is here" is a statement you must verify by reconciliation, never one the lake makes (Reconciliation).
- No ordering. Key names sort lexicographically; that says nothing about the order events happened or arrived (Event Time).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The cheapest useful check on a raw prefix is a partition completeness sweep: for every expected period, assert a partition exists, its file count is within its own historical band, and its row count is not zero. This catches a stopped writer, a half-run, and a source outage in one query (Volume Anomalies).
- It misses everything about the contents. A partition with the right file count and a plausible row count can be entirely null, entirely duplicated, or entirely last week's data re-landed under today's key.
- It also misses type drift between partitions, because the sweep never reads a value. Add a schema-fingerprint check that records the observed types per partition and alerts on change (Schema Evolution).
- A lake introduces almost no latency of its own — a write is visible about as soon as it completes. The delay a consumer feels comes from the writer's schedule and from whatever downstream job turns files into a queryable table.
- Freshness is per-prefix, and a lake makes it invisible: there is no
last_updatedon a bucket. The only honest freshness signal is the newest partition key present, and that lies whenever a writer creates the partition before filling it (Freshness Monitoring). - A lake can be extremely fresh and completely unusable at the same time. Streaming arrivals into thousands of small objects gives you second-level landing and hour-level query time (File Compaction).
- Schema-on-read means the schema changes whenever a producer feels like it and you find out at query time — usually on the first query that spans the change (Breaking Schema Changes).
- Because old partitions are immutable, evolution produces a lake where *the same dataset has several schemas across its history*. Any reader over a wide range has to reconcile them, and the reconciliation logic is business logic nobody wrote down (Semantic Changes).
- The fix is not to prevent evolution — you do not control the producer — but to detect it at landing and record it, so the reconciliation is deliberate rather than discovered during an incident (Contract Enforcement).
- A lake is the recovery position for everything downstream. If the raw zone is intact and transformations are deterministic, almost any modelling mistake costs a re-run rather than the data (Reprocessing vs Retrying).
- Recovery *of the lake itself* is much weaker. Object versioning helps against an accidental overwrite; nothing helps against a writer that has been silently landing wrong bytes for a month, because the wrong bytes are now the only bytes.
- Re-landing a period must write to a new partition or a new prefix and be swapped in, never overwrite in place — an in-place rewrite destroys the evidence you would need to explain the discrepancy (Planning a Backfill).
What can go wrong
- The swamp: prefixes with no owner, no catalog entry, no declared grain and no consumer, which cannot be deleted because nobody can prove nothing reads them.
- Small-file accumulation from a high-frequency writer, degrading every downstream read until someone schedules compaction.
- Partial partitions from a failed job, read as if complete.
- Type drift across partitions that only fails on a query spanning the boundary.
- The mitigation failing too: a catalog that is populated manually drifts from the bucket within weeks, and a stale catalog is more dangerous than none because people trust it (Metadata: Technical, Operational and Business).
- Lifecycle rules deleting a prefix that a quarterly job still depends on, discovered a quarter later (Storage Lifecycle).
- "The lake is the cheap option." Storage is the cheap part; the expensive parts are the queries the layout makes wasteful and the engineering time spent rebuilding what a database gave you (What Actually Drives Data Platform Cost).
- "We have a data lake" as a statement of capability. A bucket with files in it is a bucket with files in it. What makes it a lake is the catalog, the ownership, the layout and the contracts around it.
- "Store everything, decide later" as a permanent policy. It is a correct *landing* policy and a catastrophic *organising* policy — the deciding has to happen at some point and it gets harder every month.
- "The lake replaced the warehouse." They answer different questions. Table formats narrow the gap in transactions and performance; they do not give a lake the query planner, the workload management or the governance surface of a warehouse (Lake vs Warehouse vs Lakehouse).
- "Raw means unprocessed, so quality does not apply yet." Quality checks on raw are the cheapest checks in the platform, because they localise a fault to ingestion before it has propagated into twenty models (Data Quality).
- A lake concentrates data from every source system into one blast radius, and it does so before anyone has classified any of it. Access control at bucket granularity is far coarser than the row- and column-level control the source systems had (Data Access Control, Row and Column Security).
- Deletion is the hardest obligation to meet in a lake, because objects are immutable and there is no index from a person to the files containing them. Designing for deletion at landing — partitioning or keying so that a subject's data is locatable — is far cheaper than solving it during an audit (Deletion Requests).
- Retention is a recovery-window decision and a liability decision at the same time. Write it down per dataset, at landing, with the owner's name on it (Data Retention).
Operating it
- Object count and total bytes per prefix over time. A step change in count without a step change in bytes is the small-file problem announcing itself (File Size and the Small-Files Problem).
- Newest partition key per dataset versus its expected schedule — the closest thing a lake has to a freshness metric (Freshness Checks).
- Catalog coverage: fraction of prefixes with an owner and a description. It is a governance metric, and it is the one that predicts whether the lake will still be usable in two years (Data Discovery).
- Read amplification per query: bytes scanned versus bytes the query needed, which exposes layout problems before the bill does (Scan Cost).
- At 10x volume the lake itself is unbothered — object storage absorbs it. What breaks first is listing: a prefix with a very large number of keys makes planning slower than reading.
- At 100x, layout is the whole game. Partition choice, file size and format decide whether a query is expensive or impossible, and the lake has no optimiser to compensate for a bad one (Physical Data Layout).
- Consumer growth scales the governance problem, not the storage problem. Twenty datasets and four analysts need no catalog; a thousand datasets and eighty consumers cannot function without one (The Data Catalog).
- Retained bytes accumulate forever and nothing deletes them by default. This is the driver that grows quietly, because no single decision causes it (Storage Lifecycle).
- Per-request charges and listing calls dominate for datasets with many small objects — the cost tracks *file count*, not data volume, which is deeply counter-intuitive (Object Storage as Data Infrastructure).
- Scanned bytes at query time, decided months earlier by the partition scheme and the file format rather than by the query (Partition Pruning).
- Reprocessing: every full re-read of history to fix a bug costs proportional to all history, not to the fix (Compute Waste).
- Schema-on-read buys the ability to land sources you do not understand and costs you every guarantee a database gives about what is in the store. You are choosing to defer work, and deferred work accrues interest.
- Immutable raw retention is the strongest recovery position available and the clearest privacy liability. Those two arguments meet at retention policy and neither wins outright (Data Retention).
- A lake pushes cost from storage to query. That is a good trade when data is written far more often than it is read, and a poor one when a hundred dashboards scan the same prefix hourly (Data Marts).
Lake or swamp
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe schema-on-read property, the absence of cross-object atomicity and the swamp failure mode hold on any object store used this way. What varies is the tooling layered on top, not the underlying behaviour of a keyspace full of bytes.
- CLOUD-SPECIFICListing behaviour, consistency of listings after a write, and per-request accounting differ between providers and have changed over time; a pipeline that assumes a listing shows every object it just wrote is relying on provider behaviour rather than on a property of object storage.
- ORG-SPECIFICCatalogs, ownership and classification are coordination machinery: below a handful of datasets and one team they are pure ceremony, and above a few hundred datasets and several teams they are the only thing preventing the swamp outcome.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a store that accepts writes from anywhere cannot also offer cross-object atomicity for free, and what a listing after a write is actually promising.
- — DevOps / Production Engineering owns the access policy and lifecycle rules that make "raw is write-once" enforceable rather than aspirational.