Cost Optimization Lab
Two queries over the same dataset, doing wildly different amounts of work. Find the driver — and notice that the thing which looks expensive is rarely the thing worth fixing, because the expensive-looking thing is usually the one somebody already thought about.
Everything on this page is expressed as relative work — bytes read against bytes that had to be read, one query against another. There are no prices anywhere in this domain, on purpose: a price is a fact about somebody else's contract in some other year, and memorising one is how you end up optimising the driver that was not moving.
Data platforms do not get expensive by a decision. They get expensive quietly, one dashboard refresh and one full rebuild at a time, and the bill arrives attributed to nothing. The first useful move is almost never an optimisation — it is attribution, because a shared platform makes every cost everyone's and therefore nobody's.
Then look at the drivers below in order. In most platforms one of the first two accounts for the majority of the movement, and both respond to layout changes that take an afternoon.
The lab
Compare two queries over the same data and account for the difference. The answer is always in what the reader was obliged to touch.
Change one thing at a time: the predicate, the columns selected, the partition key, the file size. Watch which change moves the work and which changes only move your confidence.
| Metered dimension | This layout | Moved by |
|---|---|---|
| Bytes scanned | 7.1 GBsim/day | Partition pruning and column projection, multiplied together. `SELECT *` alone defeats the cheaper of the two. |
| Requests | 19Ksim/day | File count. A partition written 96 times holds 96 objects until something compacts them, and every one is listed and opened. |
| Cluster time | not modelled | Owned by Observability & Performance. A second latency model here would disagree with that one, and one of the two would be wrong on any given page. |
SELECT * defeats column pruning entirely, which is the single cheapest optimisation a columnar format offers.The five drivers
Everything a data platform charges for reduces to one of these. Naming which one moved is the entire diagnosis.
How much of a dataset a query is obliged to read in order to answer.
How much data crosses the network between tasks because a key required it.
Everything held, in every copy, for as long as it is held — raw, modelled, extracted, and the copies nobody remembers making.
Resources reserved rather than resources used — a cluster that is up, a warehouse that has not suspended, a job that holds capacity while it waits on something else.
The same computation performed again over data that has not changed.
The moves that look like savings and are not
Each of these reduces something. None of them reduces the driver that was actually moving.
- —Moving to a colder storage tier when the driver was scan volume. Storage was never the problem, and retrieval from the cold tier adds a charge of its own.
- —Shrinking the cluster when the job is bound by one skewed task. The runtime does not move; the job simply holds fewer resources for the same wall-clock time.
- —Deleting raw history to save storage, which converts every future bug from a re-run into an unrecoverable loss.
- —Compressing harder on a workload already bound by CPU, which makes the query slower and the storage line smaller.
- —Aggregating a table so nobody can query it expensively — which works, right up until somebody needs the detail that was thrown away.