The question this answers
My topic promises ordering. Ordering of what, exactly?
Records within a single partition are totally ordered by offset, and every consumer reading that partition observes them in that order. Across partitions of the same topic there is no ordering guarantee of any kind — not even approximate, and not even for records appended seconds apart.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A producer knows which partition its record was written to and the offset it received. It does not know the relative order of that record against records it sent to other partitions. A consumer knows the order of records within each partition it owns; if it owns several, the interleaving it observes between them is an artefact of fetch timing, not a fact about the stream, and treating it as one is a class of bug that only appears under load.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The split, and what it buys
A single append-only log has a hard ceiling: one leader accepting appends, one sequence, one machine’s disk and network. Partitioning breaks the topic into N such logs, each with its own leader, its own file set and its own offset space. Throughput scales with N because appends go to different machines, and consumption scales with N because partitions can be read in parallel.
The cost is paid in exactly one currency: ordering. There is no global sequence any more. Record A written to partition 0 and record B written to partition 3 have no defined order — a consumer may see either first, and different consumers may see them differently. Nothing is wrong when that happens; there was never a promise to break.
This is the single most misunderstood property of log-based systems, and the misunderstanding is systematic. Teams test with one partition, observe perfect ordering, ship, scale to twelve partitions for throughput, and discover months later that a state machine occasionally receives Updated before Created. The tests still pass, because they run with one partition.
The partition key is the ordering decision
A record’s partition is normally chosen by hashing a key: partition = hash(key) mod partitionCount. Records sharing a key therefore share a partition, and are therefore totally ordered relative to each other. That is how you buy ordering — per key, by construction, and nowhere else.
So the design question is never "do I need ordering?" but "ordering of what, with respect to what?". Every event for one user in order? Key by user_id. Every event for one aggregate in order? Key by aggregate id. Every event in the system in order? One partition, and you have given up scaling — occasionally the right answer for a low-volume control stream, and almost never right otherwise.
Choose the key too fine and you get ordering guarantees you do not need with no downside except that cross-key invariants are unenforceable. Choose it too coarse and you concentrate traffic. Choose *no* key and records are distributed round-robin, which maximises balance and gives you no ordering at all — a legitimate and underused choice for genuinely independent events.
The trap in the middle is the natural-looking business key that is not uniform. Keying by tenant_id in a B2B product means one enterprise customer with 40% of your volume gets 40% of it on one partition, and that partition’s consumer is the bottleneck for the whole group. See Hot Partitions: The Skew Hashing Cannot Fix for the general treatment and Performance’s hot-keys for the measurement problem: aggregate metrics average the saturated partition with the idle ones and report that everything is fine.
| Ordering guaranteed | Balance | Typical failure | |
|---|---|---|---|
| No key (round robin)protocol | None | Excellent | A causally-dependent pair processed out of order |
| key = user_idassumption | Per user | Good, if users are many and similar | A power user or a bot creates a warm partition |
| key = tenant_idtypical | Per tenant | Poor in B2B — tenants differ by orders of magnitude | One enterprise tenant saturates one partition permanently |
| key = order_idprotocol | Per order | Excellent | Cross-order invariants have no ordering to rely on |
| key = constantprotocol | Total order | None — one partition does everything | Throughput ceiling of a single log; group parallelism of 1 |
| key = region, then userassumption | Per region | Depends entirely on region sizes | A composite key that hides the skew inside it |
Partition count is a decision you cannot easily revisit
Partition count sets the parallelism ceiling for every consumer group on the topic: a group can never have more actively working members than partitions, so the twenty-fourth instance of a group on a six-partition topic sits idle holding no assignment. Adding partitions is therefore how you raise the ceiling — and it has a consequence that is easy to miss and expensive to discover.
The mapping is hash(key) mod N. Change N and keys move. Records for alice written before the change are in the old partition; records written after are in a different one. Per-key ordering is broken across the change, permanently for the records already written, and a consumer reading both partitions has no way to reconstruct the correct sequence. Any consumer holding per-key state — a windowed aggregate, a state machine, a deduplication cache — now has that state on the wrong partition.
There is no clean in-place fix. The workable approaches are to over-provision partitions up front (they are cheap relative to the pain — a factor of two to four above current need is normal), or to migrate by writing to a new topic with the new count and replaying, cutting consumers over once they have caught up. Some systems avoid modulo entirely by using consistent hashing so that increasing capacity moves only a fraction of keys; see The Ring: Keeping the Mapping Stable When Membership Changes and Architecture’s consistent-hashing for why that changes the arithmetic.
Partitions are not free either. Each is a set of files, a leader election candidate, replication traffic, and an entry in every rebalance computation. Thousands of partitions per broker degrade recovery time and make Rebalancing: Everyone Stops So the Partitions Can Move slower. Over-provision deliberately, not maximally.
1hash("alice") = 882132 3N = 6 -> 88213 mod 6 = 1 // all of alice's history is in partition 14N = 12 -> 88213 mod 12 = 7 // every NEW alice record lands in partition 75 6Consequences at the moment N changes:7 - partition 1 holds alice[0..k], partition 7 holds alice[k+1..]8 - the consumer of partition 7 has no per-key state for alice9 - the consumer of partition 1 holds state that will never be updated again10 - a consumer reading both cannot order across them: different offset spaces11 12The consumers do not fail. They produce wrong answers, quietly, for the13subset of keys that moved -- which is most of them.What partition-scoped ordering is actually enough for
Once you accept that ordering is per key, a productive question follows: how much correctness can per-key ordering buy? Rather more than people expect, and the answer shapes good stream design.
Most invariants people believe are global turn out to be per-entity. "An order cannot ship before it is paid" is per order. "A user’s balance never goes negative" is per user. "A device’s telemetry is monotonic" is per device. Each of these is enforceable with the right partition key and no coordination whatsoever — which is Coordination Avoidance: Restructuring the Problem Instead of Paying for It achieved through data layout rather than through protocol, and it is the single best trick in stream processing.
The invariants that genuinely span keys — "total inventory across all warehouses is non-negative" — cannot be enforced this way, and no partition key rescues them. Those need Cross-Partition Operations: Paying for What the Split Took Away, real coordination, or a redesign that makes the invariant per-key (reserve inventory per warehouse rather than globally). Recognising which kind you have, early, is worth more than any amount of later tuning.
Key points
- A topic is N independent logs. Ordering is total within a partition and completely undefined across partitions.
- The partition key *is* the ordering decision: same key means same partition means guaranteed relative order.
- A skewed key produces a hot partition, and aggregate metrics average it away with the idle ones.
- Partition count caps consumer-group parallelism, and changing it moves keys, breaking per-key ordering across the change.
- Most invariants believed to be global are per-entity, and per-key ordering enforces them with no coordination at all.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The producer computes a partition: hash of the key modulo partition count, or round-robin when no key is given.
- • The record is appended to that partition’s leader and assigned the next offset in that partition’s offset space.
- • The leader replicates to followers; the append is acknowledged once the configured replication level is met.
- • Consumers are assigned whole partitions and read each sequentially, preserving that partition’s order.
- • Offsets are meaningful only within a partition — offset 500 in partition 0 has no relationship to offset 500 in partition 1.
- • A skewed key concentrates traffic on one partition, which saturates while others idle.
- • Two causally related records are written with different keys and processed out of order.
- • Partition count is increased and per-key ordering silently breaks for keys that moved.
- • A partition leader fails; that partition is unavailable for writes until a new leader is elected, while the rest of the topic is unaffected.
- • A consumer owning several partitions interleaves them and a developer reads meaning into the interleaving.
- • Out-of-order state transitions under load: the operator sees "update for unknown entity" errors that appear only at high throughput and never reproduce locally, because local testing runs one partition.
- • Hot partition saturation: one partition’s consumer lag climbs continuously while eleven others sit at zero. Aggregate lag is unremarkable, CPU across the consumer group averages 25%, and scaling the group changes nothing because the hot partition already has a dedicated member.
- • Silent corruption after a partition-count change: per-key aggregates for most keys stop updating or restart from zero. No component errors; the numbers are simply wrong, and the change that caused it was a routine capacity increase weeks earlier.
- • Idle consumer instances: a group scaled to 24 on a 6-partition topic shows 18 members with no assignment and no throughput improvement, while the deploy is recorded as a successful scale-up.
- • Single-partition throughput ceiling: a topic keyed by a constant to "guarantee ordering" plateaus at one broker’s write capacity, and no amount of cluster growth helps.
- • Within a partition: a leader totally orders appends, which is real coordination and is what makes the ordering guarantee possible at all.
- • Across partitions: none, deliberately. That absence is exactly what buys the horizontal scaling, and exactly why cross-partition ordering does not exist.
- • Per-key ordering achieved by partitioning is coordination *avoided*, not coordination performed: the data layout makes the guarantee free rather than negotiated.
- • A partition leader failure affects only that partition; the rest of the topic continues accepting writes and serving reads.
- • Ordering within a partition survives leader election, provided the new leader is chosen from replicas that had the committed records.
- • Records acknowledged under a weaker replication setting can be lost during leader election, and the ordering guarantee says nothing about records that were never durable.
- • Detect: per-partition lag and per-partition throughput, never aggregate. Aggregate metrics are structurally unable to show partition skew.
- • Contain: for a hot partition, change the key for new writes (add a suffix to spread a single hot key) rather than adding partitions, which breaks ordering for everything.
- • Recover: for a partition-count change already made, migrate to a new topic and replay, rather than trying to repair state in place.
- • Reconcile: rebuild per-key derived state from the source of truth for any key that moved partitions.
- • Verify: per-partition lag flat across all partitions, and the key-distribution histogram roughly even.
- • Lag and throughput per partition, plotted per partition and not summed.
- • Key-cardinality and key-frequency distribution — the top ten keys by volume tell you your skew before it becomes an incident.
- • Partition count versus active consumer count per group, to catch the idle-member ceiling.
- • Producer-side partition assignment distribution, which catches a key bug before the consumer feels it.
- • Leader-election events per partition, which correlate with brief write unavailability that looks like a producer bug.
- • Throughput beyond a single log, which is the reason partitioning exists.
- • Per-entity ordering requirements, which partitioning satisfies for free with the right key.
- • Parallel consumption, where each partition is an independent unit of work with no coordination between workers.
- • Genuine global-ordering requirements, which force a single partition and eliminate the benefit.
- • Highly skewed key distributions, where a partition becomes a permanent bottleneck no scaling can address.
- • Low volume, where the operational cost of partitioning buys throughput you do not need.
- • Cross-key invariants, which partitioning cannot help with and can make harder by spreading the relevant records.
- • A single-partition topic when volume is low and total ordering genuinely matters — a control-plane or configuration stream is the classic fit.
- • Round-robin with no key when events are independent, which maximises balance and states honestly that no ordering exists.
- • Consistent hashing or explicit partition assignment instead of modulo, so capacity changes move a fraction of keys rather than nearly all of them.
- • Ordering enforced in the consumer by sequence number rather than by transport: carry a per-entity version and buffer or reject out-of-order records. Works across any transport, at the cost of consumer-side state.
A topic is not one log — ordering lives inside a partition
What people believe, and what is true
Kafka guarantees message ordering.
It guarantees ordering within a partition. Across partitions of the same topic there is none, and the difference only becomes visible after you scale past one partition.
More partitions is always better for throughput.
Partitions cost files, replication, leader elections and rebalance time. Past a few thousand per broker, recovery and rebalancing degrade noticeably.
I can add partitions to scale an existing topic.
You can, and existing keys will move. Per-key ordering breaks across the change and per-key consumer state ends up on the wrong partition.
Using a key makes distribution even.
It makes it deterministic. Evenness depends entirely on how uniform the key distribution is, and business keys are almost never uniform.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A topic is several independent logs. Records with the same key go to the same log and stay in order relative to each other. Records with different keys have no order at all.
Practical
Pick the key by asking what needs to stay in order relative to what. Check the key-frequency distribution before shipping. Over-provision partitions two to four times current need, because changing the count later breaks per-key ordering. Alert on per-partition lag, never aggregate.
Advanced
Partitioning converts one total order into a partial order, and the partial order it chooses is exactly the one your key defines. That is a design freedom, not a limitation: you are declaring which pairs of events must be comparable and accepting that all other pairs are concurrent in the sense of Happens-Before: The Only Ordering You Actually Have. Invariants that only relate comparable events are then enforceable locally, with no coordination — which is the same insight that makes CRDTs and single-key linearizability work. The bugs come from believing the partial order is a total order, and load is what makes the difference observable.
Apply it
- 🔧 Publish causally related events with different keys and measure how often the consumer sees them out of order as throughput rises.
- 🔧 Plot the key-frequency distribution of a production topic and compute what fraction of traffic the top key represents.
- ⚡ One partition’s lag climbs for a week while the other eleven are at zero. Identify the cause and the fix that does not break ordering.
- ⚡ A capacity increase from 6 to 12 partitions was made two weeks ago and per-user counters are now wrong for most users. Explain and plan the recovery.
- 💬 What ordering does a partitioned log guarantee? Be precise about the scope.
- 💬 You need every event for a user in order but the topic must scale. How?
- 💬 What breaks when you increase partition count from 6 to 12?