Parquet Internals

• A row group is a horizontal cut of the table: a set of complete rows, stored column by column within the cut. Its size is chosen by the writer, and it is the unit at which the file can be split across workers (Partitions: the Unit of Parallelism).
• Inside a row group, each column's values form a column chunk — a contiguous byte range. The footer records that range's offset and length, which is what lets a reader fetch two columns out of forty with two byte-range requests rather than a full read (Projection Pushdown).
• A column chunk is divided into pages, and the page is where encoding and compression are actually applied. A dictionary page holds the chunk's dictionary; data pages hold encoded values. Each page carries its own header with its own value count and, in newer page versions, its own statistics.
• The footer holds the schema and, per column chunk, the offset, the encodings used, compressed and uncompressed sizes, and statistics { min, max, null_count }. Those statistics are written by the writer from the values it saw, which is why they are trustworthy — and why a writer that skips them leaves nothing for the reader to work with.
• Skipping is a range-intersection test, nothing more sophisticated. The engine takes the predicate's implied range and checks it against each chunk's min/max. Disjoint means skip; overlapping means read. This is why the whole mechanism is a function of sortedness: sorting is what makes the ranges disjoint (Partition Pruning).
• Nesting is stored with definition and repetition levels: two small integer streams alongside each leaf column that encode, respectively, how deep the non-null path went and where a repeated element started a new list. A nested record is therefore still stored as flat leaf columns, which is how the format keeps columnar benefits for JSON-shaped data (Ingestion: Parsing & Chunking).

SCHEMA
  order_id        int64                       required
  customer        group                       optional
    ├── id        int64                       required
    └── country   binary (utf8)               optional
  items           group (LIST)                repeated
    └── element   group
        ├── sku   binary (utf8)               required
        └── qty   int32                       required

LEAF COLUMNS ACTUALLY STORED
  order_id
  customer.id           + definition levels
  customer.country      + definition levels
  items.element.sku     + definition levels + repetition levels
  items.element.qty     + definition levels + repetition levels

WHAT THIS COSTS
  - one physical column per leaf, not per top-level field
  - level streams per optional/repeated leaf, paid for EVERY row
    including the rows where the value is absent
  - statistics quality on leaves varies by writer and engine
  - the file's "row" is an order; an item-level aggregate needs an
    explode, and the grain changes at that point

• Choose the sort key from the predicate you actually run, then verify with a query profile that row groups are being skipped. A sort order that does not show up as skipped row groups bought nothing (Clustering and Sort Order).
• Set the row group size against predicate selectivity. If your typical filter selects a small slice, smaller row groups skip better; if you mostly full-scan and aggregate, larger groups compress better and cost less to plan.
• Confirm statistics are being written, per column, by reading a file's footer. This is a one-line check with a library and it is the first thing to look at when pruning fails.
• Filter on stored columns rather than on expressions over them. Materialise event_date as a column if you filter on the date of a timestamp, instead of casting at query time (Predicate Pushdown).
• Keep string columns you filter on reasonably short. Min/max statistics on very long strings are truncated by writers, and truncated bounds are wider than real ones, which weakens pruning without any visible signal.
• Treat nested structures as a cost. They work, they preserve columnar layout, and they make statistics, pruning and evolution all harder to reason about — flatten what consumers query on (Model Layering).

• Statistics, when present, are a sound bound: every value in the chunk lies within the recorded min and max. A reader may therefore skip on disjointness without risk of dropping a matching row. They are not tight — a chunk with one outlier has a range as wide as that outlier.
• Null counts are exact when written, which makes IS NULL and IS NOT NULL predicates prunable in a way that value predicates on unsorted data are not.
• Statistics are optional. A file with none is a valid Parquet file, and its absence is silent — there is no error, only a query that reads everything.
• Row group boundaries guarantee row completeness: a row is never split across row groups, so a worker handed a row group has whole records and needs nothing from its neighbours.
• Nothing guarantees that two files in the same dataset used the same row group size, the same encodings or the same statistics settings. Physical uniformity is a property of your writer, not of the format.