Lakes, Warehouses & Lakehouses

Object storage, analytical warehouses, and the table-metadata layer that gave files transactions. Compared on data types, query patterns, governance, cost, openness and tooling — not on marketing.

The Data Lake
▶ lab

One place to land structured, semi-structured and unstructured data before anyone knows which questions it will answer — and the reason most lakes become swamps.

Q · Where do you put data whose schema you do not control, at a volume a database will not hold, before you know what it will be used for?
The Data Warehouse
▶ lab

An analytical database built for large scans and aggregations over structured, modelled data — and what it gives you that a pile of files cannot.

Q · What does an analytical database do that a query engine over files does not, and what are you paying for it in flexibility?
The Lakehouse
▶ lab

Object storage for the bytes, a table metadata layer for the transactions, and independent query engines on top — a combination rather than a product.

Q · Can files on object storage behave like tables, and what exactly is added to make that true?
Open Table Formats
▶ lab

How Iceberg, Delta and Hudi turn immutable objects into a transactional table: a manifest of which files are the table now, and a commit that is one pointer swap.

Q · How can a set of immutable files on storage with no rename and no multi-object atomicity behave like a table with transactions?
Lake vs Warehouse vs Lakehouse
▶ lab

A comparison across data types, query patterns, governance, cost shape, openness, transactions and tooling — with the vendor framing removed and the overlap admitted.

Q · Given this data, these consumers and this team, which storage posture actually fits — and why is the honest answer so often "more than one"?
Object Storage as Data Infrastructure
▶ lab

Buckets, keys, objects and metadata — and the four properties of that model that decide how every data pipeline above it must be written.

Q · What does it change about a pipeline that its storage has a flat key space, no rename, per-request cost and immutable objects?
Data Marts
▶ lab

A narrow, purpose-built, usually pre-aggregated serving copy that trades flexibility and freshness for query cost and simplicity.

Q · When is it worth materialising a narrower copy of a model, and what does every consumer of that copy lose?
Separating Storage from Compute
▶ lab

Scale each independently, point many engines at one copy, pay for compute only while it runs — and pay a network, a cold start and the loss of locality for it.

Q · What changes when the machine holding the data is not the machine querying it, and what does that separation cost?