Salting a Skewed Key
Split the dominant key into several artificial sub-keys so its rows land in several partitions, then combine the partials. It works, it costs an extra stage — and applied to every key instead of the hot one, it does nothing at all.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
One key holds most of the rows and I cannot change the grain or broadcast the other side. How do I make that work divisible?
The same schedule that the straggler was threatening. Salting is a technique with no user-visible output: done well, a consumer sees only that the table arrives on time; done badly, they see an extra stage on the bill and the same arrival time (Straggler Tasks).
Salting operates on the key, replacing k with (k, salt) for the duration of the shuffle and then collapsing back to k. The output grain is unchanged — that is the requirement — and the intermediate grain is deliberately finer.
Add a random suffix to the group-by key so the rows spread out, then strip it off at the end. It is a three-line change and it is the right idea; what matters is which keys it is applied to and how the partials are recombined.
Salting every key uniformly. Splitting each of eight keys into six multiplies the partition count and the mean by six, so the largest partition is still the same multiple of the mean and the straggler is exactly as long. The in-repo model asserts this: with no hot key, salting is a no-op rather than a false win (Data Skew).
- Salting every key uniformly. Splitting each of eight keys into six multiplies the partition count and the mean by six, so the largest partition is still the same multiple of the mean and the straggler is exactly as long. The in-repo model asserts this: with no hot key, salting is a no-op rather than a false win (Data Skew).
- Salting a non-combinable aggregate. A sum of partial sums is a sum; an exact
COUNT(DISTINCT)of partial distinct counts is not a distinct count, and the second pass returns a confidently wrong number (Aggregation: COUNT, SUM, AVG, GROUP BY, HAVING). - Salting one side of a join and forgetting to replicate the other. Rows whose salt does not match simply do not join, and the result loses rows silently — a correctness bug introduced by a performance fix (Missing Rows).
- Hard-coding the hot key. The dominant tenant changes, the salted key is now an ordinary one, and the real hot key is unsalted and back to being a straggler.
- Leaving the salt in the output. The published table now has a grain nobody expects and a column that means nothing outside the job that created it (Grain: What Does One Row Represent?).
What is actually happening
- The partitioner sends rows by
hash(key) % partitionCount. Replacing the key withkey || '#' || (n mod S)gives one key S distinct hashes, so its rows spread across up to S partitions instead of one (The Shuffle). - Aggregation then happens in two passes: a first aggregation on the salted key, which is now evenly spread, and a second on the original key over the much smaller set of partials. The second pass is cheap because it operates on S rows per original key rather than on all of them (Parallel Reduce).
- For a join, the mechanism is asymmetric. The large skewed side gets a salt; the other side must be replicated S times, once per salt value, so every salted variant of the key still finds its match. That replication is the join's version of the cost.
- The reason uniform salting fails is arithmetic rather than incidental. Multiplying every key by S multiplies the largest partition and the mean by the same S, and the ratio between them — which is what sets the stage duration — is unchanged (Straggler Tasks).
- The in-repo model makes exactly this distinction: it finds the dominant key first, and salts only if that key holds more than a third of the rows. Its test asserts both halves — salting materially reduces the straggler ratio on a skewed run, and leaves it identical on an even one.
Two passes, and the arithmetic that has to hold
The technique is short enough to write in one query, and the interesting part is not the syntax but the condition it depends on. A salted aggregation computes partials on a finer key and then combines them on the original key, which is only equal to the unsalted result when the aggregate can be combined that way.
Sums, counts, minimums and maximums combine trivially: the sum of partial sums is the sum. Averages combine if you carry the sum and the count separately and divide at the end. Approximate distinct sketches combine by design, which is one of the underrated reasons to use them. Exact distinct counts, medians and arbitrary percentiles do not combine at all, and salting them produces a plausible wrong number.
The second requirement is that the salt is derived deterministically. A random salt makes a retried task assign rows differently from its first attempt, which is exactly the non-determinism that makes retry unsafe — so derive it from a column instead (Determinism: Same Input, Same Output?).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Output has exactly one row per original key | The salt was removed and the published grain is unchanged. | A salt column left in the final GROUP BY, which multiplies the row count by the salt factor. | A correct row count with wrong values, which is what a non-combinable aggregate produces. |
| Salted result reconciles with an unsalted run over a closed period | The two-pass computation is equal to the one-pass one. | Exactly the non-combinable aggregate case — an exact distinct count or a median that was salted. | Nothing about performance. A job can reconcile perfectly and still have gained nothing from the salt. |
| max/median shuffle-read bytes per task, before and after | The salt actually flattened the distribution it was applied to. | Uniform salting, which leaves this ratio unchanged while adding a stage. | A straggler whose cause was the machine rather than the data, where this ratio was never the problem (Straggler Tasks). |
| Row count of the salted side versus the source for the period | A salted join did not lose rows. | Incomplete replication of the non-salted side, where unmatched salts drop rows. | Duplicated rows from over-replication, which needs a uniqueness check on the business key instead (Duplicate Rows). |
1-- Step 0: find the hot key. Salt this one, not all of them.2-- country = 'US' holds ~80% of rows in the teaching model.3 4-- Pass 1: aggregate on the salted key. This is the shuffle that was skewed.5WITH salted AS (6 SELECT country,7 MOD(order_id, 6) AS salt, -- deterministic, not random8 SUM(amount) AS part_amount,9 COUNT(*) AS part_orders10 FROM orders11 WHERE dt = DATE '2026-08-25'12 GROUP BY country, MOD(order_id, 6) -- 'US' now spans 6 partitions13)14-- Pass 2: combine the partials on the original key. Small: 6 rows per country.15SELECT country,16 SUM(part_amount) AS revenue,17 SUM(part_orders) AS orders18FROM salted19GROUP BY country;20 21-- Combines correctly: SUM, COUNT, MIN, MAX, AVG (as SUM/COUNT),22-- approximate distinct sketches23-- Does NOT combine: exact COUNT(DISTINCT), MEDIAN, PERCENTILE24-- -- these need every row of a key in one placeSalting only the hot key keeps pass 2 tiny. Salt every country and pass 2 grows by the salt factor for no reduction in the largest partition — which is the mistake the next section is about.
Why salting everything moves nothing
This is the part that is genuinely counter-intuitive and the reason the in-repo model tests for it explicitly. Salting looks like it spreads work, so spreading *more* work should be better. It is not, and the arithmetic is short enough to check on paper.
Take eight keys with one at 80% of the rows. Eight partitions, mean of one eighth, largest of 0.8 — a ratio of about 6.4. Now salt every key into six buckets. There are forty-eight partitions, the mean is one forty-eighth, and the largest is 0.8 divided by six. The ratio is 6.4 again. Both numbers were divided by the same factor, so their quotient did not change, and the stage duration follows the quotient.
Salt only the hot key instead and the picture changes completely: thirteen partitions — six for the salted key and seven ordinary ones — a mean of about one thirteenth, and a largest of 0.8 divided by six. The ratio falls to under two, which is the balanced case for practical purposes.
The general rule is worth stating plainly: relative imbalance is what sets stage duration, and a transformation applied uniformly to every key cannot change a relative imbalance. Only an asymmetric transformation — one that treats the hot key differently — can.
Append a random or modulo salt to every grouping key, unconditionally, because it is simpler and has no special case. Partition count rises by the salt factor, the second aggregation grows by the salt factor, and the largest partition remains the same multiple of the mean as before.
Compute each key's share on the source, apply the salt only to keys above a threshold — the in-repo model uses roughly a third of the rows — and leave every other key untouched. The second aggregation stays small because it only has to recombine the keys that were split.
Stage duration follows the ratio of the largest partition to the mean, not the absolute size of either. A uniform transformation scales both by the same factor and leaves the ratio invariant; only an asymmetric one moves it. This is asserted in scripts/de-sim.test.ts, which requires salting to materially reduce the ratio on a skewed run and to leave it exactly unchanged on an even one.
8 keys, one holding 80% of rows. [SIMULATED: arithmetic on src/de/sim/pipeline.ts] NO SALT 8 partitions mean = 12.5% largest = 80.0% ratio = 6.4x <- straggler SALT EVERY KEY x6 48 partitions mean = 2.1% largest = 13.3% ratio = 6.4x <- unchanged! SALT HOT KEY x6 13 partitions mean = 7.7% largest = 13.3% ratio = 1.7x <- fixed Dividing the largest and the mean by the same number leaves their ratio alone. Only treating the hot key differently changes anything.
Salting is the fourth choice, not the first
Because it adds a stage, a synthetic column and a correctness condition, salting should be reached for after the cheaper remedies have been ruled out — and they frequently have not been.
A composite key is free when the finer grain is still what the consumer wants. A broadcast join eliminates the shuffle entirely, so there is no partition to be skewed. Splitting a single well-known dominant entity into its own job gives predictable duration with no clever arithmetic at all. Only when none of those applies does salting earn its complexity.
And before any of them: check that the hot key is a real entity. A partition dominated by null or by a sentinel default is a data defect, and salting it distributes a meaningless group very efficiently across the cluster (Data Quality).
What is true about the grain, the other side of the operation, and the hot key?
when The hot key is null, an empty string or a sentinel default.
cost A conversation with the producing team and a test that fails on unexpected sentinel volume. It is the only remedy that removes the problem rather than accommodating it (Data Contracts).
when A finer grain — key plus day, key plus hour — is still meaningful for the output.
cost The output grain changes, so downstream aggregates must be checked against the new one (Grain: What Does One Row Represent?).
when It is a join and one side is small enough after filtering to fit in each executor.
cost A size assumption that ages, and a driver that has to assemble the broadcast (Broadcast Joins).
when The dominant key is one stable, well-known entity.
cost Two runs and a union: more predictable, more moving parts, and it needs orchestration (Orchestration).
when The key is genuinely dominant, the grain must not change, and no side is small enough to broadcast.
cost An extra stage, more shuffle bytes, a synthetic column, and an aggregate that must be verified to combine (Reduction Ordering: The Sum Changed When the Worker Count Did).
when Your engine version can split oversized shuffle partitions at runtime and the operation is one it supports.
cost Depends entirely on version and configuration, applies to some operations and not others, and cannot be relied on for a plain aggregation on a dominant key.
Adaptive skew handling — the engine detecting an oversized shuffle partition from runtime statistics and splitting it — exists in recent Spark versions for certain join types and is enabled by default in some distributions. Which operations it covers, and whether it is on in your deployment, is a version-specific question: verify it in the documentation for the version you run rather than assuming skew is handled for you.
How to build it
Most important first.
- Identify the hot key from the data rather than from memory: a count per key ordered descending, run on the source, and a threshold on its share (Data Skew).
- Salt only the keys above that threshold. Every other key keeps its natural value and its natural partition, which keeps the second pass small and the plan comprehensible.
- Choose the salt count from how far the hot key is above the mean. Enough buckets to bring it near the mean, and no more — every extra bucket is more rows in the second aggregation for no reduction in the maximum.
- Use a deterministic salt derived from a column — a row id modulo S — rather than a random number, so the job is reproducible and a retried task produces the same assignment (Determinism: Same Input, Same Output?).
- Verify the recombination arithmetic explicitly. Sums, counts, minimums, maximums and bounded sketches combine; exact distinct counts, medians and percentiles do not (Reduction Ordering: The Sum Changed When the Worker Count Did).
- Try the cheaper remedies first. A composite key that is meaningful, or a broadcast join, removes the skew without adding a stage or a concept (Broadcast Joins).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Salting guarantees the same result as the unsalted aggregation only for aggregates that combine associatively. That condition is the whole correctness argument and it must be checked, not assumed.
- It guarantees a more even distribution for the salted key and nothing about the others. Keys that were already balanced are unaffected, which is the point.
- It guarantees nothing about the output layout. Salting changes which rows are computed together, and if a downstream reader depended on the previous file arrangement, that dependency is now broken (Bucketing).
- For joins, correctness is guaranteed only if the non-salted side is replicated across every salt value. Omit that and the join silently drops rows.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Reconcile the salted result against an unsalted run on a bounded sample or a closed period. This is the check that catches a non-combinable aggregate having been salted, which is the failure that produces wrong numbers rather than slow ones (Reconciliation).
- Assert that the output has exactly one row per original key. A salt left in the grouping is visible immediately as a row count that is S times too large (Data Tests).
- Both miss a join where the replication was incomplete, if the missing matches happen to be rare. Pair them with a completeness check against the source row count (Missing Rows).
- Salting converts a stage whose duration is set by the dominant key into one set by the mean, plus the cost of a second aggregation. In a badly skewed job that is a large improvement in arrival time.
- In an unskewed job it is a pure addition: an extra stage, an extra barrier, and no reduction anywhere (Stages and Tasks).
- The second pass is small — S partials per original key — so it adds a stage rather than a proportional amount of work. That asymmetry is why the technique is worth the complexity when skew is genuine.
- Which key is hot changes over time. A salting implementation that hard-codes the key is correct on the day it ships and silently useless later — the code still runs, the straggler comes back (Semantic Changes).
- Changing the salt count changes the intermediate partitioning and nothing about the result, provided the recombination is still correct.
- Adding a new aggregate to a salted query is the dangerous edit: the existing sums combine, and the newly added exact distinct count does not (Aggregation: COUNT, SUM, AVG, GROUP BY, HAVING).
- Salting is stateless and reversible: remove it and re-run. Nothing persists that needs repairing.
- If a salted job published wrong numbers because a non-combinable aggregate was salted, the recovery is a backfill of the affected range with a corrected query — and the wrong numbers are already in dashboards (Planning a Backfill).
- A deterministic salt makes a re-run produce the identical intermediate arrangement, which is what makes debugging a salted job tractable (Determinism: Same Input, Same Output?).
What can go wrong
- Uniform salting: more partitions, identical straggler ratio, extra stage. The job is slower and the report says it was optimised.
- A non-combinable aggregate salted, producing a fast wrong answer that passes every structural check.
- A salted join missing its replication, dropping rows that had no matching salt.
- A hard-coded hot key that has since stopped being hot, so the technique is paid for and no longer applies.
- The mitigation failing: raising the salt count to chase a residual straggler, which grows the second aggregation without touching the cause when the cause was actually a slow machine (Straggler Tasks).
- "Salting spreads the load." It spreads the load of the key it is applied to. Applied uniformly it spreads everything equally and improves nothing (Data Skew).
- "More salt buckets are better." Past the point where the hot key is near the mean, extra buckets only enlarge the second aggregation.
- "Salting is a performance change, so it cannot affect correctness." It changes the grouping, so it changes any aggregate that is not associative. That is a correctness change wearing performance clothing.
- "We salted, so skew is handled." Skew is handled for the key that was hot when the code was written. Which key is hot is data, and data moves (Volume Anomalies).
Operating it
- Max versus median shuffle-read bytes per task, before and after. This is the number salting exists to move, and if it did not move, the change did nothing (Tail Latency: Why p50 Being Fine Does Not Help).
- Stage count and total shuffle bytes, so the cost of the second pass is visible alongside its benefit (Pipeline Metrics).
- The identified hot key and its share, logged per run. It is the input to the decision and it changes over time, so it belongs in the run's metadata rather than in someone's memory.
- At 10x, a genuine hot key stays hot and the technique keeps working; the salt count usually needs to grow with the imbalance rather than with the data.
- At 100x, salting is often not enough on its own and the hot key is handled as a separate job, which gives predictable duration and independent scheduling (Orchestration).
- As key cardinality grows, the second aggregation grows with it — S partials per key — so a very high-cardinality salted aggregation eventually has an expensive second pass.
- An extra shuffle and an extra barrier for the second aggregation, paid on every run whether or not the skew is present that day (The Shuffle).
- For joins, replication of the non-salted side by the salt factor — a real multiplication of bytes moved, which is why broadcasting is preferable when it is available.
- Complexity cost in the transformation itself: a reader now has to understand why a synthetic column exists and why the aggregate is computed twice (dbt Concepts).
- Salting buys an even distribution with an extra stage, extra shuffle bytes and a transformation that is meaningfully harder to read.
- It also degrades output locality: rows for one key were computed in several partitions, so the resulting files are less clustered by that key than they would otherwise be (Clustering and Sort Order).
- It is the remedy of last resort among the skew fixes, and it is still the right answer when the grain cannot change and no side is small enough to broadcast.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALSplitting a hot key into sub-keys and recombining is a technique for any partitioned system, including sharded databases and partitioned logs where a hot key overloads one shard. What differs is whether recombination is a second query, a second stage or a fan-in at read time.
- SIMULATEDThe claim that uniform salting does not move the straggler ratio comes from
src/de/sim/pipeline.ts, which salts only a key holding more than about a third of the rows, and fromscripts/de-sim.test.ts, which asserts that salting an even distribution leaves the ratio exactly unchanged rather than appearing to help. - ENGINE-SPECIFICRecent Spark versions can split an oversized shuffle partition at runtime for some join types, which covers part of what manual salting is for; Flink and the distributed warehouses expose no equivalent knob and either rebalance internally or leave the rewrite to you. Check what your engine version actually does before hand-rolling this.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the general hot-key problem — one key overwhelming one shard — and the fan-out/fan-in patterns used to spread and recombine it in serving systems rather than in batch jobs.