Platforms & Cloud Services

The primitives first, then how BigQuery, Snowflake, ClickHouse, DuckDB and the managed streaming services realise them — architecturally, not from a feature list.

Cloud Data Services
▶ lab

A data platform is assembled from about seven primitives. Every cloud sells all seven under different names, and the names are the least interesting part of the comparison.

Q · Your platform runs on one provider and someone asks what the equivalent service is on another. What does "equivalent" have to mean before that question has an answer?
Comparing Analytical Warehouses
▶ lab

Six axes that actually separate analytical warehouses — architecture, storage/compute coupling, latency profile, concurrency model, cost-model shape and workload fit — and why a product name is the last thing to decide.

Q · Three teams recommend three different warehouses. What are you comparing them on, before anyone runs a benchmark?
BigQuery Concepts
▶ lab

A serverless analytical engine: columnar storage you do not manage, compute allocated per query rather than provisioned, and exactly two physical knobs — partitioning and clustering — carrying all the layout weight.

Q · There is no cluster to size, no node to add and no index to create. What is left that decides how much a query reads and how long it takes?
Snowflake Concepts
▶ lab

Three separated layers — immutable columnar storage, independently sized compute clusters, and a services layer that holds all the metadata — and what that separation actually buys, which is isolation and elasticity rather than speed.

Q · Which of a warehouse's three jobs — holding bytes, running queries, and knowing what exists — decides whether a query reads most of a table or almost none of it?
ClickHouse Concepts
▶ lab

A columnar OLAP database built for logs, events and real-time aggregates: immutable sorted parts merged in the background, a sparse index over granules, and a sort order that decides almost everything.

Q · You need aggregates over billions of event rows fast enough that a human keeps clicking rather than waiting. What makes that possible, and what did you agree to give up for it?
DuckDB Concepts
▶ lab

An analytical database that runs inside your process. No network in the hot path, no cluster, no concurrency story — and that combination changes what a pipeline stage costs, not just how fast a query is.

Q · What changes about a data pipeline when the analytical database is a library in your process rather than a service across the network?
Managed Streaming Platforms
▶ lab

Managed Kafka, Kinesis, Pub/Sub and Event Hubs are all realisations of the same primitive — a durable, replayable log. The axis on which they genuinely differ, and the one that changes your design, is ordering.

Q · Four managed services all give you a durable stream that many consumers can read independently. Which of their differences will actually force you to change your pipeline?
Choosing an Analytical Platform
▶ lab

Eight questions that turn "which warehouse should we use" into a list of required capabilities, two or three candidate architectures, and the trade-off each one asks you to accept. The output is never a single product.

Q · Someone asks which analytical platform to adopt. What do you have to know before that question has an engineering answer rather than a preference?