File Compaction
Rewriting many small files into fewer larger ones. What it buys, what it costs, and what a reader sees if it is halfway through when their query starts.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The table is made of far too many files. Rewriting them into fewer, larger ones fixes the reads — so what does that rewrite cost, and what happens to everyone reading the table while it runs?
Two consumers with opposite interests. The reader wants few large sorted files and does not care when they were written. The writer wants to commit as soon as it has data and does not care how many objects that makes. Compaction is the scheduled negotiation between them, and it is a third party that both must tolerate.
The unit is a compaction group: the set of files selected to be rewritten together, almost always bounded to one partition so that the rewrite is local and its blast radius is one directory rather than the table.
Notice the table has too many files, run a job that reads a partition and writes it back with fewer output files, and delete the originals. It works the first time you do it by hand on a quiet Sunday, which is exactly why it becomes a scheduled job without anyone thinking through the concurrency.
A long-running query planned against the old file list, then goes to read a file the compaction job has already deleted. The query fails with a missing-file error that has nothing to do with the data and everything to do with timing.
- A long-running query planned against the old file list, then goes to read a file the compaction job has already deleted. The query fails with a missing-file error that has nothing to do with the data and everything to do with timing.
- A worse variant: the job deletes the originals *after* writing the new files but the metadata is not updated atomically, so for a window the table contains both. Every row in the compacted range is now duplicated, every count is wrong, and every check that only looks at freshness passes (Duplicate Rows).
- Compaction and the streaming writer touch the same partition at the same time. The writer commits a file the compaction job had already read but not accounted for, and that file is deleted along with the originals (Atomic Publish).
- The job compacts the current, still-open partition. Data arriving during the rewrite is either lost or duplicated depending on which order the steps happened to run in.
- Compaction runs less often than fragmentation accumulates. File count trends up across months while the job reports success every night — a maintenance process losing a race it appears to be winning (The Pipeline Succeeded. The Data Is Wrong.).
- The rewrite "helpfully" cleans as it goes — deduplicating, casting a column, dropping a field nobody uses. The original bytes are deleted and the transformation is now unauditable and unrepeatable (Keeping Raw History: The Recovery Position and the Liability).
What is actually happening
- Compaction is a read-rewrite-swap: select a set of files, read their rows, write them out as fewer and larger files, then change the table's definition of which files it consists of. The third step is the one that decides whether the operation is safe.
- With a plain directory-based table, the table's definition is "whatever objects exist under this prefix", so the swap is a delete — and deletes are not atomic with respect to a reader that already listed the directory. This is the source of every concurrency failure in the lesson (The Data Lake).
- With a manifest-based table format, the definition is a metadata file, so the swap is a single commit that points at the new files. Readers hold a snapshot; a reader that started before the commit keeps reading the old files, which still exist, until it finishes (Open Table Formats).
- That is why the old files cannot be deleted at commit time. They are garbage only once no reader can still be holding a snapshot that references them, which makes expiry a separate, delayed operation rather than part of the rewrite (MVCC: Multi-Version Concurrency Control).
- Compaction is also the natural moment to establish sort order, because the rows are being rewritten anyway. Sorting during compaction costs almost nothing extra and turns a file-count fix into a skipping improvement (Clustering and Sort Order).
- The same primitive appears one layer down in storage engines: an LSM tree merges small sorted runs into larger ones for exactly these reasons, and pays exactly this write amplification for it (LSM Trees: Why Some Engines Favour Writes, Compaction: The Merge That Pays for Cheap Writes).
Read, rewrite, swap, expire
Compaction has four steps and most implementations get the first two right. The value of writing them out as stages with explicit guarantees is that it makes visible how much of the operation's safety lives in the third step, which is the one that looks like bookkeeping.
Note where the guarantees weaken. Selection and rewriting promise a lot and are easy to verify. The swap promises only what the underlying mechanism provides — and under a plain directory layout, that is nothing. Expiry promises safety only if it waits, and waiting is the step most likely to be tuned down by someone looking at a storage chart.
One property should be preserved across all four stages and is worth stating as an invariant: the set of rows visible to a reader never changes. Every failure mode in this lesson is a violation of that invariant, and every check worth writing is a test of it.
- 1Select
Chooses a bounded set of files, normally within one closed partition, that are below the target size.
guarantees The blast radius is one partition, and partitions still receiving writes are excluded.
fails by Selecting an open partition, so files arriving during the rewrite are neither included nor protected.
- 2Rewrite
Reads every selected row and writes it out as fewer, larger — ideally sorted — files in a new location.
guarantees Row-level preservation: the output contains exactly the input rows, and no transformation was applied.
fails by Cleaning, casting or deduplicating on the way through, which makes the operation unverifiable and the original unrecoverable.
- 3Swap
Changes the table's definition of which files it consists of, from the old set to the new.
guarantees Only what the mechanism provides. A table-format commit gives atomicity and a stable snapshot for in-flight readers; a directory delete gives neither.
fails by A window in which both sets are visible (duplication) or neither is (missing data), observed by whichever query happens to plan inside it.
- 4Expire
Deletes the pre-compaction files once no reader can still reference them.
guarantees Storage is reclaimed without breaking a reader — but only if the delay exceeds the longest possible reader lifetime.
fails by Running too soon, which both breaks in-flight readers and destroys the rollback position at the same time.
Read the guarantees column. Three of the four stages promise something structural; the swap promises only what you built underneath it, which is why the choice of table format is really a choice about this one row.
What a reader sees while it runs
The interesting question is not what compaction does to the table. It is what a query that started thirty seconds before the swap observes. There are only three possible answers and two of them are wrong.
The correct answer is that the reader continues against a consistent set of files — either entirely the old set or entirely the new — for the duration of its query. Achieving that requires that a reader's view is pinned at plan time and that the files it pinned still exist when it reads them, which is two separate requirements and most naive implementations satisfy neither.
The two wrong answers are the failure modes worth naming precisely. Duplication happens when both sets are visible at once: every row in the compacted range is counted twice, and because the row count went up rather than down, most volume monitoring interprets it as a busy day. Missing files happen when the old set is deleted while a reader holds it: this one at least fails loudly, which makes it the better of the two.
The comparison below is the entire design decision, and it is not really about compaction — it is about whether the table has a definition separate from what happens to exist in a directory.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A query plans against the old file list, then the swap deletes those files. | Intermittent query failures naming a file that does not exist, unreproducible minutes later. | No snapshot isolation: the reader's file list and the table's file list are separate things that diverged. | Delay expiry beyond the longest reader lifetime; if the format supports snapshots, pin the reader to one. |
| New files are written before the old ones are deleted, with no atomic commit between. | Row counts and revenue in the compacted range double, then return to normal after the deletes finish. | Both file sets were part of the table simultaneously. | Make the swap a single commit. Until then, always write-then-commit-then-expire and never write-then-delete (Atomic Publish). |
| Compaction runs against the partition a streaming job is still writing to. | A handful of records that were definitely ingested are absent from the table. | Files that arrived after selection and before the swap were not in the new set and were deleted with the old one. | Only compact closed partitions, gated by a watermark; if the format supports it, let the commit fail on conflict and retry (The High-Water Mark). |
| Expiry is tightened to reclaim storage. | Rollback from a bad compaction is impossible; reader errors reappear. | The retention window was doing two jobs — reader safety and rollback — and was tuned against neither. | State the window explicitly as the maximum of longest-reader and rollback-decision time, and set the storage lifecycle from that (Storage Lifecycle). |
| The compactor is given a sort order and later a deploy removes it. | File sizes stay healthy; queries filtering on a non-partition column get slower with no other change. | Compaction restored file size and destroyed clustering, so statistics stopped being selective. | Treat sort order as part of the table definition, and assert on the read side by tracking the pruning ratio (Clustering and Sort Order). |
Write the new files into the same prefix, then delete the originals. The table is "whatever objects exist under `events/date=2026-08-25/`", so the change takes effect as the individual object operations land.
Write the new files to a new location, then make one atomic commit that changes the table's manifest from the old file set to the new one. The pre-compaction files remain on storage and are deleted by a separate, delayed expiry job.
A directory has no atomic multi-object operation, so the interval between the first write and the last delete is a window in which the table is either duplicated or incomplete, and any query planning inside that window observes a state that never existed as a consistent whole. Moving the table's definition into a metadata file makes the transition a single commit and lets in-flight readers keep a stable snapshot — and separating expiry from commit is what turns an irreversible rewrite into one you can roll back.
When compaction is the wrong answer
Compaction is a repair. It is the correct response when a producer must legitimately write small and often, and the wrong response when the producer is fragmenting for no reason — in which case fixing the writer prevents the problem instead of rewriting its output forever.
It is also wrong on cold data. The trade is compute now against read savings later, so on a range that will be read twice more in its life the rewrite costs more than every future query it improves. A table with a long tail of rarely-touched history should have a compaction policy with a horizon, not one that reaches back to the beginning.
The most expensive mistake is not choosing wrongly between these options; it is choosing one and never revisiting it. A schedule that was right when the writer committed hourly is wrong the day the writer moves to per-minute commits, and nothing about the job will report that.
Paid once per range compacted. This is write amplification and it is the entire cost side of the trade (Write, Read and Space Amplification).
The main saving, and it recurs. Its total value is this weight multiplied by how many times the range will be read again — which is why cold data fails the test.
Free to acquire during a rewrite that is happening anyway, and worth nothing if queries do not filter on the sort column.
Both file sets held for a bounded period. The cheapest line here and the one most often removed first.
Grows with partition count rather than data volume, and becomes significant on a table that is over-partitioned (Partition Cardinality).
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights illustrating the shape of the trade, not measurements. The decision rule they encode: compaction pays when the recurring saving, multiplied by the number of future reads, exceeds the one-off rewrite — so it is nearly always right for hot data and nearly always wrong for archives.
What is producing the fragmentation, and who is going to read the result?
when A batch job over-parallelises its output, or a connector writes one file per page for no operational reason.
cost A change to a producing job and possibly reduced write parallelism. Strictly better than compacting, because it prevents rather than repairs (File Size and the Small-Files Problem).
when A streaming or micro-batch writer must commit frequently, and the data is queried regularly after it stops changing.
cost A second job, its compute, a retention window of duplicated storage, and a concurrency surface. The default answer for a hot lake table.
when The table is fragmented and queries also filter on a non-partition column.
cost A sort during the rewrite — nearly free given the read and write are already paid. Skipping this because "we only wanted file size" leaves the cheapest improvement on the floor (Clustering and Sort Order).
when A historical backlog is fragmented but the writer has since been fixed.
cost A large one-off rewrite. Correct, and it should not become a recurring schedule, because the recurring cost buys nothing once the backlog is gone.
when The range is cold, queried rarely, and the rewrite would cost more than every remaining query against it.
cost Those rare queries stay slow. Usually the right trade, and worth writing down so that the next person to notice the file count does not "fix" it (Cost vs Freshness).
How to build it
Most important first.
- Bound every compaction to a single closed partition, and select within it only the files below the size threshold. Compacting an open partition is the source of most of the hard cases and is usually avoidable by waiting; re-compacting partitions that are already in good shape costs proportional to history and buys nothing (The High-Water Mark).
- Make the swap atomic. If the table format offers a commit, use it. If it does not, write to a new directory and swap the pointer a consumer resolves — never delete-then-write, and never write-then-delete without a commit in between (Atomic Publish).
- Separate expiry from commit. Retain the pre-compaction files for at least as long as your longest-running reader, then delete them as a distinct scheduled operation with its own metric.
- Sort during the rewrite, since the read and the write are already being paid for. This is the cheapest sort order you will ever buy (Clustering and Sort Order).
- Never transform during compaction. It is a physical operation and must remain byte-preserving at the row level, so that "did compaction change the data?" always has the answer "no, by construction" (Reconciliation).
- Alert on the rate — file count per partition over time — rather than on the job's exit status, because a compaction job that is losing to ingest succeeds every single night (Pipeline Metrics).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Compaction guarantees, when done correctly, that the set of rows is unchanged. That is the only guarantee that matters and it is the one an incorrectly implemented compaction breaks by duplicating or dropping rows.
- Atomicity of the swap is guaranteed only by whatever mechanism performs it. A table format's commit gives it; a directory rewrite gives nothing, and a reader can observe an intermediate state that never existed as a consistent whole.
- Snapshot isolation for in-flight readers is a table-format property, not a compaction property. Under a plain directory layout there is no snapshot, so a reader's file list can go stale mid-query and there is no mechanism that prevents it.
- Nothing guarantees ordering between compaction and a concurrent writer unless the format provides conflict detection at commit. Two processes writing the same partition without it is a lost update wearing a maintenance job's clothes (MVCC: Multi-Version Concurrency Control).
- Compaction guarantees nothing about correctness of the data. It preserves whatever was there — including duplicates, which it makes larger and better organised (Deduplication).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that must exist: row count and a summed measure before and after, per compaction group, compared and alerted on any difference. Compaction is byte-preserving by definition, so any divergence is a bug and there is no acceptable variance to tune away.
- A second check on the read side: query the compacted range for duplicate business keys immediately after the swap. This catches the both-old-and-new-files window, which the count check can miss if the count was taken before the failure (Data Tests).
- What both miss is a compaction that lost and gained the same number of rows, and anything about a partition the job did not select. They also say nothing about whether the *layout* improved — a compaction that ran and produced the same number of files passes every correctness check and achieved nothing.
- Compaction does not change freshness. The newest row is exactly as new after the rewrite as before, which is worth saying because "we compacted" is sometimes offered as an answer to a staleness complaint.
- It does change the *variance* of query latency: a partition swings between fragmented and compacted, so the same query over the same day is materially cheaper after the job runs than before it. Consumers experience that as intermittent slowness with no cause they can see.
- The compaction lag — how far behind the newest partition the compactor is — is the number that describes this, and it deserves to be a published metric rather than an implementation detail (Freshness Monitoring).
- A schema change between the original write and the compaction is the interesting case: the rewrite must produce files that are readable under the table's current schema without changing what old rows meant. A compactor that re-writes old rows under a new schema has performed a silent migration (Schema Evolution).
- Column addition is usually safe — old files gain the column as null on read. Type changes are not: a compaction that casts an old column to a new type has rewritten history and there is no original left to compare against (Breaking Schema Changes).
- Because compaction rewrites file boundaries, it also rewrites statistics. A table whose statistics were selective can become less selective if the compactor concatenates without sorting — the same rows, worse skipping.
- The recovery position is the pre-compaction files, which is exactly why expiry must be delayed rather than immediate. With them retained, an incorrect compaction is undone by pointing the table back at them (Rolling Back Data).
- With a manifest-based format this is a metadata operation: roll the table back to the snapshot before the commit. That is the strongest argument for such a format that does not involve the word "transaction" (Open Table Formats).
- Once the originals are expired there is no recovery from a compaction that lost rows, because the rewrite was the only copy of the arrangement and the rows are gone. This is the one way a physical operation becomes a data-loss event (The Raw Landing Zone).
What can go wrong
- Both old and new files visible simultaneously, duplicating every row in the compacted range while every freshness and volume check reports normality.
- A reader holding a stale file list encountering a deleted object, surfacing as an intermittent query failure that is unreproducible by the time anyone investigates.
- Compaction racing a streaming writer for the open partition, dropping the files that arrived during the rewrite.
- Expiry that is too aggressive, removing the only rollback position before anyone has checked the result.
- A compaction job that is scheduled correctly, succeeds correctly, and cannot keep up — the failure of the mitigation rather than of the system.
- A rewrite that silently loses sort order, restoring file size while removing the skipping the table used to have (Clustering and Sort Order).
- "Compaction is just an optimisation, so it cannot break anything." It rewrites and deletes data. A compaction bug is a data-loss bug with a maintenance job's reassuring name.
- "We compact nightly, so file count is under control." File count is controlled when compaction outpaces fragmentation. Nightly against a per-minute writer may or may not, and the exit status will not tell you (Pipeline Metrics).
- "Compaction deduplicates." It does not, unless you made it, and making it means it is no longer byte-preserving and no longer trivially verifiable. Deduplication is a transformation and belongs in a transformation (Deduplication).
- "Delete the old files as soon as the new ones are written." That removes the rollback position and, without an atomic commit, exposes readers to a table that is briefly missing part of itself.
- "Bigger compaction runs are more efficient." They amortise setup better and they widen the blast radius, lengthen the window in which a concurrent writer can conflict, and make a failed run expensive to retry.
Operating it
- Files per partition before and after each compaction run, which turns "did it work" into a number instead of an exit code.
- Compaction lag: how many recent partitions remain uncompacted. The derivative of this is the metric that catches a job losing the race (Pipeline Metrics).
- Row count delta per compaction group, alerting on anything other than zero. The single highest-value check in this lesson.
- Bytes rewritten per run versus bytes scanned saved on the read side, tracked together — this is the ratio that says whether the maintenance is paying for itself (Scan Cost).
- Reader errors referencing missing files, which are almost always a compaction concurrency signal rather than a storage one.
- At 10x ingest, compaction must run more often rather than bigger, because the backlog per partition grows and a single enormous rewrite has a correspondingly enormous blast radius.
- At 100x, compaction becomes continuous rather than scheduled, and the interesting question changes from "when do we run it" to "how do we prevent it competing with ingest for the same resources".
- At any scale, the number of *partitions* to consider grows the cost of deciding what to compact, independently of data volume — a selection step that lists everything becomes its own small-files problem (Partition Cardinality).
- Compaction costs a full read and a full write of everything it touches. That is write amplification, and it is paid every time a range is compacted rather than once (Write, Read and Space Amplification).
- Compacting the same range repeatedly is the classic waste: a schedule that re-compacts already-compacted partitions costs proportional to history and buys nothing (Compute Waste).
- It buys reduced request count and reduced scan cost on every subsequent read. The trade is only worth it when reads outnumber rewrites, which is why rarely-queried tables should be compacted rarely or not at all.
- Retaining pre-compaction files for a rollback window costs storage for both copies during that window — cheap, and the cheapest insurance in this lesson (Storage Lifecycle).
- Compaction spends compute to save compute. It is only a good trade when the saved read cost, summed over all future queries against that range, exceeds the rewrite — which means it is close to always right for hot data and close to always wrong for cold archives.
- A safe compaction requires either a table format with atomic commits or a swap discipline you build and maintain yourself. The first is a dependency; the second is a source of subtle concurrency bugs.
- Delaying expiry to keep a rollback window costs duplicated storage and adds a second scheduled job. Not delaying it costs the ability to undo, which is worse.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- FORMAT-SPECIFICIceberg, Delta and Hudi make the swap an atomic metadata commit and give in-flight readers a stable snapshot, so the concurrency failures here become conflict-detection questions. A plain directory of Parquet files has no commit and no snapshot, so the same operation is genuinely racy and must be made safe by convention.
- ENGINE-SPECIFICEngines differ in whether they re-list files during a long query or plan once at the start, which changes how long a reader can hold a stale file list and therefore how long pre-compaction files must be retained before expiry is safe.
- WAREHOUSE-SPECIFICManaged warehouses perform an equivalent reorganisation internally and do not expose the files, so the concurrency problems here are the vendor's rather than yours — but so is the timing, and you cannot compact on demand before a known-expensive query.
- GENERALThe read-rewrite-swap shape and its write amplification are the same primitive an LSM storage engine runs continuously. What changes between contexts is the size of the unit and who is responsible for atomicity, not the trade being made.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what "atomic commit" means when the metadata store and the object store are different systems, and why a two-step swap across them cannot be made atomic by trying harder.
- — DevOps / Production Engineering owns how a maintenance job like this is deployed, scheduled and rolled back, and why a job whose success metric is its exit code is a job nobody is monitoring.