Parquet
A self-describing, columnar, splittable file whose footer tells a reader what it can skip — which is a different claim from "it is smaller".
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What does a Parquet file physically contain that a compressed CSV does not, and which of those things does a query actually use?
Every analytical reader of a data lake: Spark and Trino jobs, DuckDB on a laptop, a warehouse's external table, a training pipeline pulling features. All of them read the same bytes through different libraries, which is the property that makes an open format worth choosing at all (Open Table Formats).
One file holds many row groups; one row group holds one column chunk per column; one column chunk holds many pages. The row group is the grain that matters most, because it is simultaneously the unit of parallel read, the unit of statistics and the unit of skipping.
Adopt Parquet because everyone says to, write it with default settings, and treat it as "the fast format". This mostly works — the defaults are sensible and the format is genuinely good — and it is why a lot of teams cannot explain why one of their tables is fast and another is not.
A pipeline writes one Parquet file per micro-batch. There are now hundreds of thousands of small files, each with a footer to read and a task to schedule, and query planning takes longer than the scan (File Size and the Small-Files Problem).
- A pipeline writes one Parquet file per micro-batch. There are now hundreds of thousands of small files, each with a footer to read and a task to schedule, and query planning takes longer than the scan (File Size and the Small-Files Problem).
- Analysts write
SELECT *. Column pruning — the largest advantage the format offers — is switched off by the query text, and the scan costs what a row format would have cost (Projection Pushdown). - A predicate on a column the data is not sorted by prunes nothing, because every row group's min/max spans the whole domain. The format is working exactly as designed and the layout is wrong (Clustering and Sort Order).
- A team writes one row group per file with a very large row group size, so a query that needs a thousand rows still reads the whole group. Skipping granularity was set by a config value nobody revisited.
- A column's type changes upstream from
int32tostringbetween two runs. Both files are valid Parquet, the directory now has two schemas, and reads either fail or silently null depending on the engine's merge behaviour (Schema Evolution).
What is actually happening
- A Parquet file is self-describing: the schema lives in the file, in the footer, alongside the metadata. There is no external header, no separate DDL and no ambiguity about types — a reader that has the file has everything it needs (Metadata: Technical, Operational and Business).
- It is columnar within a horizontal slice. The file is first cut into row groups (a set of complete rows), then each row group is stored column by column. That combination is what lets a reader both prune columns and parallelise across row groups; a purely column-major file with no horizontal cuts would give the first and not the second (Row vs Column Storage).
- The footer is read first. It sits at the end of the file with a length and a magic number, so a reader seeks to the tail, reads the metadata, and from that learns the schema, the row group boundaries, the byte offset of every column chunk, and the statistics for each. Only then does it issue reads for the byte ranges it actually needs (The Parquet Read Path).
- Per-chunk statistics — min, max, null count, and optionally distinct count — are what turn metadata into skipping. If a row group's
event_daterange does not intersect the predicate, the entire group is never read. This is predicate pushdown at the file level, and it is a property of the file rather than of the engine (Predicate Pushdown). - Compression is applied per page, inside a container that remains splittable at row group boundaries. That is the structural difference from a gzipped CSV: both are compressed, only one can be read in parallel and only one can be read partially (Why Analytical Data Compresses).
- Nested data is handled by definition and repetition levels, which encode where nulls and list boundaries occur without materialising a tree per row. This is why Parquet handles nested JSON-shaped records without abandoning the columnar layout, and it is also the part most people never look at (Parquet Internals).
What is inside the file
The structure is small enough to hold in your head, and holding it is what makes every layout decision in this domain explainable. Four levels: the file, the row groups inside it, the column chunks inside each row group, and the pages inside each chunk. Then a footer at the end that maps all of it.
The order matters. Because the file is cut horizontally *first* into row groups and only then column-major inside each one, a reader gets both properties it needs: it can read one column without touching the others, and it can hand different row groups to different workers. A file that was purely column-major end to end would be a single unsplittable unit.
The footer sitting at the *end* is not an accident either. A writer does not know the byte offsets or the statistics until it has written the data, so the map can only be produced last. That is the same reason a truncated Parquet file is unreadable rather than partially readable — the map never got written.
events-2026-08-25.parquet
│
├── Row Group 0 (a horizontal slice: complete rows)
│ ├── Column Chunk: event_ts ├─ pages ─ encoding ─ compression
│ ├── Column Chunk: country ├─ pages ─ dictionary ─ compression
│ ├── Column Chunk: revenue ├─ pages ─ encoding ─ compression
│ └── Column Chunk: session_id └─ pages ─ plain ─ compression
│
├── Row Group 1
│ ├── Column Chunk: event_ts
│ ├── Column Chunk: country
│ ├── Column Chunk: revenue
│ └── Column Chunk: session_id
│
├── ... more row groups ...
│
└── FOOTER <- read FIRST, seek from end of file
├── schema (names, types, nullability, nesting)
├── per row group: row count, total byte size
├── per column chunk: byte offset, length, encodings,
│ compressed + uncompressed size,
│ statistics { min, max, null_count }
├── key/value metadata (writer version, created_by, custom keys)
└── footer length + magic bytes "PAR1"Against a compressed text file
The comparison people reach for is CSV, and the useful version of it is not about size. Both files can be compressed. Only one of them lets a reader answer "which bytes do I actually need" before reading anything.
Work through a concrete query: SELECT country, SUM(revenue) FROM events WHERE event_date = DATE '2026-08-25' GROUP BY country, against a table with forty columns. The Parquet reader consults the footer, learns it needs two column chunks per surviving row group, checks event_date statistics to eliminate groups, and issues byte-range reads for what remains. The CSV reader has no choice but to decompress and parse every byte of every row, then discard thirty-eight fields per row.
The parse cost is the part usually left out. CSV parsing is character-by-character work with quoting rules and type inference on every field; a Parquet column chunk is a typed, length-prefixed byte range that decodes without any of that (Vectorized Execution).
Thirty-eight of forty column chunks are never fetched. This is the dominant effect and it disappears entirely the moment someone writes SELECT *.
Proportional to how disjoint the per-chunk min/max ranges are, which is a direct function of sort order. On randomly ordered data this driver collapses to near zero.
Text parsing is per character with quoting rules; typed decoding is per value over a length-prefixed range. Larger the wider the rows and the more string columns there are.
Real CPU work that a columnar format adds back. It is the cost of the saving above it, and a heavier codec grows this bar specifically.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a narrow projection over a wide table — an ordering, not a measurement. Invert the query to SELECT * with no predicate and the first two bars vanish, which is precisely why the format is not "fast" on its own.
The reader must decompress the entire stream because the compressed unit is the whole file, parse every field of every row to find the two it wants, infer types on the way, and cannot split the file across workers. Nothing in the file says what is in it, so a wrong assumption about a column's type is discovered as a cast error mid-scan.
The reader seeks to the footer, reads the schema and offsets, eliminates row groups whose `event_date` range misses the predicate, issues reads for exactly two column chunks in the survivors, and decodes typed values with no parsing. Different row groups go to different workers.
The difference is not compression — both are compressed. It is that Parquet stores a map of its own contents, so the reader can decide what to read; and it stores values grouped by column and by type, so the reader can decode without parsing. A text format has no map and no types, so every read is a full read regardless of what the query asked for.
What Parquet deliberately does not do
A large share of production confusion comes from expecting the file format to provide things that live a layer above it. Parquet describes one file. It has no concept of a table, a transaction, a version, a schema authority or a concurrent writer.
This is a good design decision rather than a gap. A file format that stayed out of the table-management business is why the same files are readable by Spark, Trino, DuckDB, pandas and a warehouse external table without any of them agreeing on a catalog. But it means every guarantee people attribute to "our Parquet lake" is actually being provided — or not provided — by something else (Open Table Formats).
The decision table below is the one to have before writing the first file, because retrofitting a table format onto a directory of Parquet is considerably more work than starting with one.
What does this dataset need beyond "bytes a reader can interpret"?
when A single writer, append-only, partition-per-day, and consumers who read yesterday rather than the partition currently being written.
cost No atomicity, no snapshot isolation, no safe concurrent write, no schema enforcement across files. Every one of those is your pipeline's job.
when One writer per partition and consumers who must never see a half-published partition.
cost Write to a staging path and move on completion. Simple and effective; still no schema enforcement, and object storage rename semantics vary by provider (Object Storage as Data Infrastructure).
when Concurrent writers, row-level updates or deletes, time travel, or a schema that must evolve safely across many consumers.
cost A metadata layer to operate and understand, a catalog to run, and a new class of maintenance work — snapshot expiry, manifest rewrites, compaction (Open Table Formats).
when The team wants none of the above operational surface and the data does not need to be readable by arbitrary engines.
cost You give up open access to the bytes and take on the warehouse's model of storage and compute. Often the right call, and rarely presented as a trade (The Data Warehouse).
Which encodings, logical types, column-index structures and bloom-filter options a given Parquet library writes and reads changes across versions, and engines differ in which of them they use for pruning. Treat the four-level structure and the footer contract as stable and verify current library documentation for anything more specific.
How to build it
Most important first.
- Size files so each one is a sensible unit of work rather than a unit of arrival. Write fast and small on the hot path if you must, then compact to larger files on a schedule (File Compaction).
- Select the columns you need. Every other lever in this lesson is secondary to not reading columns you will not use, and it costs nothing but discipline (Projection Pushdown).
- Partition on the coarse filter (usually a date) and sort or cluster within the partition on the next most common predicate. Partitions eliminate directories; statistics eliminate row groups inside the surviving directories (Partitioning).
- Set row group size deliberately. Larger groups compress better and give coarser skipping; smaller groups skip more precisely and add footer overhead. Pick from your predicate selectivity, not from a blog post.
- Enforce a schema at the write boundary. Parquet will happily let two files in one directory disagree, and the resulting read behaviour differs by engine — which is a contract problem, not a format problem (Contract Enforcement).
- Put a table format on top once you need atomic commits, snapshots or safe concurrent writes. Parquet is a file format and has no opinion about which files constitute a table (Open Table Formats).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The file is self-describing and immutable once written. A reader needs no external catalog to interpret it, and any reader implementing the spec reads it identically.
- Statistics, where present, are truthful about the values in that chunk — a writer that records min/max is asserting a bound the reader may rely on to skip. Statistics are optional, though, and a file without them is valid and simply prunes nothing.
- There is no transaction, no atomicity across files, and no notion of a table. A directory of Parquet files being read while another job writes into it has no consistency guarantee whatsoever; that is what table formats were invented to add (Open Table Formats).
- No ordering is guaranteed to a reader. Rows come back in file order in practice, and a query that depends on that is depending on an implementation detail that a compaction will change (Atomic Publish).
- Nothing guarantees schema consistency across files in a directory. That is enforced by whatever wrote them, or not at all.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that matters here is a schema-consistency assertion across the files of a table: read the footer schema from a sample of files per partition and assert they are identical, including types and nullability.
- Add a null-rate check per column against its own history. A column that silently becomes all-null after an upstream rename produces perfectly valid Parquet with perfectly valid statistics (Data Tests).
- Both miss values that are present, well-typed and wrong. Parquet's self-description is about structure, and structure being correct is not evidence that the numbers are (Trusting Data).
- Parquet is a batch-shaped format. A row group must be buffered before it can be written, so a writer trades between flushing small groups often (fresh, poorly encoded, many files) and buffering longer (stale, well encoded, fewer files).
- This is the structural reason streaming pipelines land in a row-oriented format or a log first and convert to Parquet later — the conversion is a batch step by nature (Streaming Ingestion).
- A consumer reading a partition while it is still being appended to sees an inconsistent set of files unless a table format or an atomic swap provides the boundary (Atomic Publish).
- Adding a nullable column is safe: older files simply lack it, and readers project null. This is the everyday case and it works (Backward Compatibility).
- Removing a column is safe for readers that do not reference it and immediately breaks those that do. Renaming is a remove plus an add, and it breaks everything, because Parquet identifies columns by name unless a table format maintains stable field ids (Breaking Schema Changes).
- Retyping is the dangerous case. A directory containing both
int32andstringversions of a column is valid, and what happens on read is engine-specific: some merge and cast, some fail, some return null for the mismatched files (Semantic Changes). - None of this is enforced by the format. Schema evolution safety in a Parquet-based lake comes from the table format or the contract above it, and a team that believes the format is protecting them is mistaken (Data Contracts).
- Rewriting a Parquet dataset — recompacting, re-sorting, re-encoding — is a read-write-swap and is safe if the swap is atomic and the output is validated first.
- A truncated file (a writer that died before the footer was written) is unreadable, not partially readable, because the footer holds the map. This is a useful property: a half-written Parquet file fails loudly rather than returning half a partition (Partial Failure).
- Recovery from a bad write means re-deriving from raw. There is nothing inside a Parquet file that lets you repair it, which is another argument for an immutable landing zone (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- Small-file proliferation from a streaming or per-batch writer, where planning cost exceeds scan cost.
- Schema drift across files in one directory, producing engine-dependent read behaviour rather than a clean error.
- Row group size mismatched to predicate selectivity, so skipping is available in principle and useless in practice.
- Statistics absent or disabled by a writer, silently removing pushdown for every consumer.
- A reader library too old for a newer encoding or a newer logical type, which fails at read time rather than at write time — the failure lands on the consumer, not on the producer.
- A concurrent write into a directory being read, producing a partial view that no check will flag because every individual file is valid.
- "Parquet is just compressed CSV." The compression is the smallest part of the difference. Typed self-description, per-chunk statistics, column pruning and splittability are the features, and none of them is achievable by compressing a text file (CSV, JSON and Their Limits).
- "Parquet makes queries fast." Parquet makes *skipping possible*. Whether a query skips anything depends on the columns selected, the partition layout and the sort order, all of which are your decisions.
- "Parquet gives us ACID." It gives you a file. Atomic commits, snapshot isolation and safe concurrent writes come from a table format layered on top (Open Table Formats).
- "We are on Parquet so schema evolution is handled." Parquet permits evolution; it does not police it. Two files in one directory may disagree and the format is entirely comfortable with that (Data Contracts).
Operating it
- File count and file size distribution per partition. The clearest early warning of a layout problem and the cheapest to collect (File Size and the Small-Files Problem).
- Bytes scanned versus bytes stored per query, from engine statistics — the ratio that tells you whether pruning is working (Scan Cost).
- Row groups read versus row groups skipped, exposed by most engines' query profiles. If nothing is skipped, either the predicate does not match the sort order or statistics are missing.
- Footer schema fingerprints per partition, so drift is visible as a change rather than discovered as a failed query (Schema Registry).
- At 10x data, row group and file sizing decisions that were invisible start to set job runtime, because task count and planning time are now on the critical path.
- At 100x, partitioning and clustering decide whether a query is possible at all; the format is a constant and the layout is the variable (Physical Data Layout).
- At 100x *consumers*, schema stability and a table format matter more than any physical tuning, because the failure mode shifts from slow queries to concurrent writers corrupting each other's view (Open Table Formats).
- Bytes scanned dominates, and it is controlled by columns selected, partitions pruned and row groups skipped — in that order (What Actually Drives Data Platform Cost).
- File listing and planning cost scales with file count, and becomes the dominant cost on a table with many small files regardless of how little data those files hold.
- Write-side CPU covers encoding, statistics gathering and compression; the sort that makes them effective is the expensive part (Compute Waste).
- Storage is compressed size and is the smallest and most predictable line.
- Parquet is excellent for scans and poor for record-at-a-time access. Fetching one row by key means reading a row group and reconstructing it from every column chunk — a shape an operational store handles in a fraction of the work (OLTP vs OLAP).
- Immutability makes readers safe and updates expensive. Changing one row means rewriting a file, which is why merge and upsert are whole subjects here rather than statements (Upserts and Merges).
- Self-description makes the format portable and makes schema drift the writer's entire responsibility. The format will never stop you writing an inconsistent directory.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- FORMAT-SPECIFICRow groups, column chunks, pages and a tail footer are Parquet's specific structure. ORC expresses the same ideas as stripes, streams and row-index entries with a different statistics granularity, and Arrow is an in-memory layout with no file footer at all.
- ENGINE-SPECIFICWhether an engine uses page-level statistics, column indexes or bloom filters when they are present varies considerably between Spark, Trino, DuckDB and warehouse external-table readers, so the same file can prune well in one engine and not in another.
- GENERALThe underlying idea — cut the data horizontally, store each slice column by column, and put a map at the end so a reader can skip — predates Parquet and is shared by every serious analytical format. That part transfers even if Parquet does not.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what "atomic across many files in object storage" can and cannot mean — the guarantee a table format is built to provide sits on primitives that domain defines.