Parquet vs Avro
Not a rivalry. One is built for reading a few columns across many rows, the other for handling whole records one at a time — and most pipelines use both, in that order.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
For this specific hop, does the consumer read a few columns across many records, or every field of a few records at a time?
Two different consumers, which is the entire answer. The analytical consumer scans columns and wants pruning and statistics. The record consumer — a stream processor, a sink, a replay job — needs whole records and gains nothing from a columnar layout (Who Actually Consumes This Data).
The grain of the *access pattern*, not of the data. Ask what one unit of work reads: if it is "this column, across everything in range", the answer is columnar. If it is "this record, entirely", the answer is row-oriented. The same dataset can have both, at different hops.
Pick one format for the whole platform to keep things simple. Fewer libraries, fewer conversions, one mental model. This is genuinely appealing and it is why most platforms that made the choice made it for consistency rather than for fit.
Avro everywhere: the analytical layer deserialises whole records to read two fields, on every query, forever. No pruning, no statistics, no skipping (The Parquet Read Path).
- Avro everywhere: the analytical layer deserialises whole records to read two fields, on every query, forever. No pruning, no statistics, no skipping (The Parquet Read Path).
- Parquet everywhere: the streaming producer must buffer a row group before it can write anything, so a format designed for batch is now on the critical path of an event pipeline (Streaming Ingestion).
- Parquet on the broker: there is no schema registry equivalent in the picture, so producers and consumers coordinate schema by convention and a rename breaks everyone silently (Schema Registry).
- A team converts events to Parquet immediately on arrival and discards the Avro, then discovers a transformation bug and has no record-level original to replay from (Keeping Raw History: The Recovery Position and the Liability).
- The slogan "Avro for writes, Parquet for reads" gets applied to a batch job that reads every column of every row — where the columnar format's advantages do not apply and the conversion step was pure cost.
What is actually happening
- The difference is physical adjacency. In Parquet, values of one column are contiguous, so reading a column is a contiguous read and reading a record means gathering from every column chunk. In Avro, values of one record are contiguous, so the reverse holds. Everything else follows from this one fact (Row vs Column Storage).
- Columnar adjacency is what makes per-column statistics, column pruning and effective encoding possible. None of those has anywhere to live in a row format, because there is no per-column unit to describe (Parquet Internals).
- Row adjacency is what makes record-at-a-time processing cheap and streaming writes possible. A producer can emit one record with no buffering; a columnar writer cannot emit anything until it has a group's worth (Why Analytical Data Compresses).
- Schema handling differs in kind rather than degree. Parquet carries a schema per file and enforces nothing across files. Avro carries a writer schema and applies resolution rules against a reader schema, which is what makes evolution a mechanical question with a registry able to enforce it (Schema Evolution).
- The reason most real pipelines use both is that the two access patterns genuinely occur at different hops. Events arrive one at a time and are consumed one at a time; the same events are later scanned by column across months. Converting between the formats is not indecision, it is the pipeline doing its job (ETL: Transform Before the Data Lands).
- The decision degrades to "either is fine" more often than the discussion suggests. For a modest dataset queried a few times a day by one team, the format is not the constraint and picking the one your tooling handles best is the correct engineering answer (Measure Before You Optimize).
One question, asked per hop
The comparison becomes tractable the moment you stop asking it about the platform and start asking it about a single arrow in the pipeline. For that arrow: what does one unit of work on the consuming side read?
If the answer is "a few fields, across an enormous number of records", the consumer wants columns. If it is "every field of this record, then the next one", the consumer wants rows. That is the whole criterion, and everything else in this lesson is elaboration on it.
The decision below lists the four common hops in an event-to-analytics pipeline. Notice that a normal platform ends up with different answers at different rows, and that this is not a compromise — it is what having a pipeline means (The Fundamental Data Journey).
For one unit of work downstream, is the access pattern a column scan or a whole record?
when Producers emit events continuously; consumers process them one at a time; schemas evolve independently across teams.
cost A registry to operate and depend on; binary payloads needing tooling; anyone querying the topic directly reads every field (Avro).
when The zone exists to be a replay source and an audit of what actually arrived.
cost Storage for a copy that is rarely queried. It buys the ability to rebuild everything downstream, which is what makes transformation bugs recoverable (The Raw Landing Zone).
when Queries project a few columns and filter across long ranges of history.
cost A conversion step with its own schedule and freshness lag; batch-shaped writes; poor record-at-a-time access (Parquet).
when An application reads whole entities by key, or a dashboard reads pre-aggregated columns.
cost Answer it with the same question rather than by inheriting the analytical layer's format. By-key whole-record access is not a columnar workload, even inside a data platform (Data Marts).
The same dataset, both ways
Two access patterns against the same hundred million order events. One is an analyst's aggregate: two columns, filtered by date. The other is a replay: every field of every event in a range, fed to a stream processor to rebuild state.
The matrix makes the asymmetry concrete. Neither format wins both rows, and neither loses both. What changes between the rows is nothing about the data — only what the consumer asked for.
The last two rows are the ones that decide real architectures more often than the first two, because they are about operations rather than about bytes: who can read this without a registry, and where does a breaking schema change get caught.
The largest line in almost every platform, and the reason the conversion step exists. Driven by columns projected and partitions and row groups skipped.
Once per record, proportional to volume. Grows sharply if implemented as a full reconversion of history rather than incrementally (Incremental Processing).
Per record, on every consumer. Compact binary encoding keeps this low; a text format raises it on both bytes and parse cost.
Storage for the row-oriented replay source alongside the columnar tables. The cheapest line and the one that buys recoverability.
Small in resources and non-trivial in operational attention — it is a service on the producer write path with its own availability story.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a typical event-to-analytics platform, to establish an ordering rather than a magnitude. The teaching is that the conversion step people hesitate over is small against the scan cost it removes.
| Aspect | Parquet (columnar) | Avro (row-oriented) | Why it differs |
|---|---|---|---|
SELECT country, SUM(revenue) over a year | Reads two column chunks per surviving row group; skips partitions and groups by statistics. | Deserialises every field of every record in range, then discards all but two. | Column adjacency is what makes a partial read possible. A row format has no per-column byte range to fetch. |
| Replaying every event to rebuild state | Reconstructs each record by gathering from every column chunk — the format working against the access pattern. | Reads records contiguously, one after another, exactly as stored. | Record adjacency. The same property that makes column scans expensive makes whole-record reads cheap. |
| Writing from a continuous producer | Must buffer a row group before anything can be flushed, so latency has a floor set by group size. | Can emit a single record immediately; a container file writes blocks as it goes. | Columnar layout requires a horizontal slice to exist before it can be written column-major. |
| Schema evolution safety | Permitted, unenforced. Files in one directory may disagree; read behaviour is engine-specific. | Writer-to-reader resolution rules, enforceable by a registry in CI before deployment. | Avro was designed around resolution; Parquet was designed around scanning. Each solved the problem it was built for. |
| Interpretable without external services | Yes — the schema is in the footer of every file. | In a container file yes; on a broker no, because only a schema id travels with the payload. | The registry is an operational dependency people frequently forget they took on (Schema Registry). |
| Where a breaking change is caught | At read time, by a consumer, in production, engine-dependently. | At registration time, in CI, by a compatibility check. | This is the strongest practical argument for a schema-carrying format on the transport, and it is not a performance argument at all. |
Where the slogan is actually wrong
"Avro for writes, Parquet for reads" is a useful compression of the argument and it fails in two specific directions worth naming, because both occur regularly.
It fails when a read is record-shaped. A backfill that replays a year of events through a transformation reads every field of every record; it is a read, and it wants the row format. Teams who took the slogan literally have converted to Parquet and then read every column of it, paying the reconstruction cost for nothing.
It fails when a write is column-shaped. The output of a daily aggregation job is a batch of complete rows written once; there is no streaming producer and no latency floor to worry about. Writing that to a row format because "writes use Avro" gives an analytical table nobody can scan efficiently.
The criterion that does not fail is the access pattern of the consumer, one hop at a time. It is a longer sentence than the slogan and it produces the right answer in the cases the slogan gets wrong (Who Actually Consumes This Data).
This job writes, so it writes Avro. This job reads, so it reads Parquet. The rule is memorable, requires no investigation, and produces the wrong answer whenever a read is record-shaped or a write is batch-shaped.
Name the consumer of this output and describe one unit of its work. If that unit touches a few columns across many rows, write columnar. If it touches every field of one record at a time, write row-oriented. If both consumers exist, write both and accept the second copy.
Read and write are properties of the job you are looking at; column scan and record access are properties of the consumer, and only the second determines which physical adjacency helps. Substituting the first for the second is what makes the slogan fail exactly in the cases where the decision mattered.
How to build it
Most important first.
- Decide per hop, not per platform. Transport and landing want records; the analytical layer wants columns; the conversion between them is a normal, cheap, scheduled step (Raw, Staging, Curated: Layers by Purpose).
- Keep the row-oriented original when it is your replay source. The ability to reprocess from records exactly as received is worth more than the storage it costs (The Raw Landing Zone).
- Convert to columnar at the boundary where data stops being a stream and starts being a table — usually the raw-to-staging step, on a schedule that matches your freshness commitment (Raw, Staging, Curated: Layers by Purpose).
- Do not put a columnar format on a message broker, and do not query a row format across a year of history. Both are the same mistake in opposite directions.
- If a single format really must serve both, be explicit about which side you are penalising and by how much, so the decision can be revisited when that side becomes the constraint.
- Where the analytical layer needs updates, snapshots or concurrent writes, the question stops being about file formats and becomes a table format question (Open Table Formats).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Parquet guarantees a self-describing file, sound per-chunk statistics when written, splittability at row group boundaries, and column-level byte ranges. It guarantees nothing across files.
- Avro guarantees a self-describing container file, splittability at sync markers, and deterministic writer-to-reader schema resolution. On a broker it guarantees only that the payload references a schema someone else must still hold.
- Neither guarantees schema consistency across a dataset by itself. Parquet defers it to whatever wrote the files; Avro defers it to a registry's compatibility policy — the difference is that one of those is a place you can enforce it (Contract Enforcement).
- Neither guarantees correctness of any kind. Both will faithfully store a well-typed, well-encoded, entirely wrong number (Trusting Data).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that spans both formats is a row-count and key reconciliation across the conversion: records in the Avro landing zone for a period must equal rows in the resulting Parquet partition, on both count and distinct business key.
- Add a type-fidelity assertion: after conversion, verify that logical types survived — timestamps still timestamps, decimals still decimals, not silently widened to double (Data Tests).
- They miss defaults becoming values. A field filled by schema resolution with its default arrives in Parquet as a real value, reconciles perfectly, and is not something that was ever emitted (Avro).
- Row-oriented writing has no buffering floor beyond a record, so it is the format shape that can be genuinely low-latency at the producer.
- Columnar writing requires a group, so it introduces a buffering interval by construction. A platform that wants both freshness and columnar reads pays for it with a conversion step and its schedule (Cost vs Freshness).
- The conversion interval is usually the dominant term in end-to-end freshness for an event-to-analytics pipeline, and it is a knob teams forget they set (The Freshness SLO).
- Avro's resolution rules plus a registry give a mechanical answer to "is this change safe" and a place to enforce it before deployment (Backward Compatibility).
- Parquet has no equivalent. Two files in a directory may disagree, and what happens on read is engine-specific — so schema safety in a Parquet lake comes from a table format or from the pipeline that writes it (Breaking Schema Changes).
- The conversion step is where the two regimes meet, and it is the right place to enforce a contract: reject records that do not conform rather than writing a Parquet file that quietly encodes the drift (Contract Enforcement).
- Replay from row-oriented records is the cleaner path: records are whole and self-contained, and a log or container file can be re-read from a position (Replay from the Log).
- Rebuilding the columnar layer from the row-oriented landing zone is a normal, idempotent operation if the conversion is deterministic and the output path is derived from the input range (Reprocessing vs Retrying).
- The reverse — reconstructing records from a columnar table — works but is not equivalent: whatever the conversion dropped, defaulted or flattened is not recoverable from the Parquet side (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- A columnar format used as the analytical *and* the transport layer, forcing a buffering delay onto the producer.
- A row format used for the analytical layer, so every query deserialises every field of every record in range.
- The row-oriented original deleted after conversion, removing the replay source that made every transformation bug survivable.
- A conversion step that silently widens or coerces types, so the analytical layer's types differ from the contract everyone agreed (Semantic Changes).
- Schema enforced in the registry and not at the conversion boundary, so a drift that the registry permitted lands in the table anyway.
- The decision made once for the whole platform and never revisited, so a hop that changed its access pattern years ago is still on the wrong format.
- "Avro for writes, Parquet for reads." A slogan that gets the common case right and hides the criterion. The criterion is whole-record access versus column scans — a batch job that reads every field of every row is a row-shaped read, and a streaming job that projects two fields from a large table is a column-shaped one.
- "Parquet replaced Avro." They were designed for different access patterns and both are actively used, usually in the same pipeline at different hops.
- "We should standardise on one format." Standardise on one format *per layer*. Standardising across layers means one layer is on the wrong one.
- "Avro is slower." Avro is slower at column scans and faster at record access. A format has a shape, not a speed (Benchmark Fallacies: Confident Numbers That Are Wrong).
Operating it
- Conversion lag: the gap between a record landing in the row-oriented zone and appearing in the columnar table. Usually the largest term in end-to-end freshness (Freshness Monitoring).
- Reconciliation result per period across the conversion, on count and on a summed measure (Reconciliation).
- Bytes scanned per analytical query, which is the number the columnar layer exists to move (Scan Cost).
- Deserialisation cost per record on the streaming side, which is the number the row layer exists to keep low (What Serialization Costs).
- At 10x, the analytical penalty of a row format becomes hard to ignore and is usually what forces the conversion step to be built.
- At 100x, the conversion step itself needs to be incremental — reconverting all history nightly is the classic waste in this shape of pipeline (Incremental Processing).
- At 100x producers, the registry and its compatibility policy matter more than either format's physical properties, because coordination has replaced I/O as the constraint (Data Contracts).
- Analytical scan cost is dominated by whether the format permits pruning at all. A row format has no pruning to offer, so this is a step change rather than a tuning difference (What Actually Drives Data Platform Cost).
- Transport cost is dominated by bytes per record, where a compact binary row encoding is well ahead of a text one and roughly comparable to a columnar one per record (Payload Size: 20KB, 200KB, 5MB).
- The conversion step costs compute proportional to volume, once per record, and is usually small against the scan cost it removes.
- Keeping both copies costs storage twice, which is the cheapest line and buys the replay position (Storage Lifecycle).
- Using both formats costs a conversion step, two libraries, and a second copy of the data. It buys the right physical layout at each hop and a replay source, which is normally a clear win and is not free.
- Using one format costs whichever access pattern it serves badly, permanently and invisibly, in exchange for a simpler platform.
- Converting eagerly gives fresh columnar data and more small files; converting on a schedule gives better files and a longer lag. Pick from the freshness commitment you have made (The Freshness SLO).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- FORMAT-SPECIFICThe comparison is really row-oriented versus columnar, with Avro and Parquet as the representative implementations. Protobuf or a compact binary row encoding sits on Avro's side of the line; ORC sits on Parquet's, so the conclusions transfer while the specifics do not.
- SCALE-SPECIFICBelow the point where an analytical query is noticeably slow or expensive, this decision changes nothing measurable and picking the format your tooling handles best is correct. It becomes decisive when scans span months of data or when producers need sub-batch write latency.
- ENGINE-SPECIFICSome engines read Avro efficiently enough for modest analytical work and some barely support it; some warehouses ingest one format natively and require a staging step for the other. That support matrix frequently decides the question before any architectural argument does.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the delivery semantics of the transport hop this comparison sits on, which is why the row-oriented side of the pipeline inherits duplicates and the columnar side inherits whatever deduplication did.