Query Engines

Engines that query data they do not own. Coordinators and workers, pushdown, vectorized execution, and the real limits of federating a query across systems.

Query Engines
▶ lab

Engines that answer SQL over storage they do not own — what that separation buys, and every guarantee it quietly hands back.

Q · What changes when the system planning your query is not the system that stores your data?
Distributed Query Execution
▶ lab

Coordinator to workers to sources to partial results to merge — and the four places a query dies that a single-node engine never has.

Q · What actually happens between submitting a SQL string and receiving the first row, when the work is spread across a coordinator and many workers?
Predicate Pushdown
▶ lab

Push the filter down to the reader so less is read at all — and learn the identical-looking query where it silently does not happen.

Q · My query filters to one day out of a year. Why did it read the whole year?
Projection Pushdown
▶ lab

Read only the columns the query needs. The cheapest optimisation a columnar format offers, and the one `SELECT *` throws away.

Q · The query needs two columns out of eight. Why did the reader fetch all eight, and what did that actually cost?
Source Pushdown
▶ lab

If the remote system can filter, aggregate or limit, do the work near the data and move less — and know exactly which of those your connector actually supports.

Q · The engine can ask a remote database for filtered, aggregated results, or it can pull the table and do the work itself. Which one is happening right now?
Federated Query
▶ lab

One SQL statement across several systems. Genuinely useful, and it gives up consistency, predictable latency, optimiser competence and control of the load you impose.

Q · If an engine can join a lake table to a production database and a SaaS export in one query, why would anyone still build ingestion?
Vectorized Execution
▶ lab

Operators that process a batch of column values per call instead of one row at a time — and why that changes what the CPU is able to do.

Q · Two engines run the same plan over the same columnar files. Why is one of them spending most of its time on work that has nothing to do with the data?
Batch and Streaming Unification
▶ lab

Modern engines let you express both with one API. The authoring surface converged; latency, state and completeness semantics did not.

Q · If the same SQL runs over a bounded table and an unbounded stream, has the distinction between batch and streaming gone away?