Consumer Groups and the Parallelism Ceiling
A group divides a topic's partitions among its instances, one partition to at most one instance. Partition count is therefore the hard ceiling on parallelism, and instances beyond it do nothing at all.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Consumer lag is growing and we have doubled the number of instances. Why has nothing changed?
The pipeline stages downstream of the group, which experience the group's throughput as their freshness. A warehouse loader that cannot keep up does not fail — it produces correct data that is progressively further behind, and every dashboard built on it is confidently reporting a stale world (Freshness Monitoring).
The unit of work assignment is one partition to one instance, for the duration of an assignment. Not one record, not one batch — a partition. Every capacity conversation about a consumer group is really a conversation about how many partitions there are and how evenly they are loaded.
Run the consumer as a normal stateless service behind an autoscaler: measure lag, add instances when it grows, remove them when it shrinks. This is the correct instinct for a queue-shaped broker and for almost every other kind of worker pool, which is exactly why it is applied here.
The topic has six partitions and the autoscaler has scaled to sixteen instances. Ten of them are running, healthy, consuming no records, and reporting perfectly normal metrics, because a partition is assigned to at most one consumer in a group (More Threads Is Not More Speed).
- The topic has six partitions and the autoscaler has scaled to sixteen instances. Ten of them are running, healthy, consuming no records, and reporting perfectly normal metrics, because a partition is assigned to at most one consumer in a group (More Threads Is Not More Speed).
- Lag is caused by one hot partition, so the group's aggregate throughput has spare capacity and the backlog still grows. The instance on the hot partition is saturated and no rebalancing moves work away from it (Data Skew).
- Every scale event triggers a rebalance, which pauses consumption across the whole group while assignments are recomputed. An autoscaler reacting to lag therefore makes lag worse in short bursts, and the loop can oscillate.
- The consumer keeps in-memory state per partition. A rebalance moves a partition to a different instance, the new owner starts from the last committed offset with empty state, and the aggregation it emits is wrong for one window rather than failing (Checkpointing).
- Two different purposes share one consumer group by accident — the same group id copied between deployments — so each instance receives a subset of the partitions and neither purpose sees all the records. Both look like they are working.
What is actually happening
- A consumer group is a name. Instances that present the same group id are members; the broker assigns each partition of the subscribed topics to exactly one member and stores one committed offset per (group, topic, partition) (Offsets and Commits).
- The assignment is the whole mechanism. Effective parallelism is `min(instances, partitions)` — an instance beyond the partition count is a member of the group with no partitions, which is a perfectly healthy process doing nothing.
- Different groups are completely independent. Two groups on the same topic each receive every record, each at their own pace, with their own offsets. That independence is what makes a log a platform substrate instead of a point-to-point connector (The Event Log).
- A rebalance recomputes assignments whenever membership changes — a deploy, a crash, a scale event, a member that missed a heartbeat. During a rebalance the affected partitions are not being consumed, so a group that rebalances often has a lower effective throughput than its instance count suggests.
- Lag is arithmetic, not a mystery: backlog grows by
arrivals − effective capacityeach interval and shrinks by the same difference when capacity wins. It is measured in records, and converting it to a time estimate requires a throughput assumption that is least reliable exactly when lag is high. - Because assignment is per-partition, an unevenly loaded topic produces an unevenly loaded group. The busiest partition sets the group's effective throughput regardless of how much headroom the rest have (Topics and Partitions).
One partition, one consumer, and everyone else waits
The assignment rule is short: within a group, each partition goes to exactly one member. Members can hold several partitions; a partition is never split. That is enough to determine everything about how a consumer group scales.
Draw it once and the ceiling becomes obvious. Six partitions and three instances means two partitions each. Six and six means one each. Six and nine means six instances with one partition and three with none — running, healthy, heartbeating, consuming nothing. The three idle instances appear in the deployment, in the cost line, and in the group's membership list; they appear in no throughput metric because they have no throughput to report.
Groups are independent of each other, and that is the property worth protecting. A second group reading the same topic is invisible to the first: different offsets, different lag, different failures. It is why you can run a full historical reprocess against production data at three in the morning without asking anybody, and why every distinct purpose deserves its own group id (Replay from the Log).
The ceiling, computed
simulateLag() in src/de/sim/stream.ts with arrivals 1200 per tick and per-instance capacity 200 per tick. These are unitless model quantities on a teaching timeline, not a measurement of any broker, and the model reports backlog in events and never a wait time. The transferable content is the shape: capacity is min(instances, partitions) × per-instance capacity, and the backlog gap is arrivals − capacity.The table below comes from the repository's own lag model. Arrivals are held at 1200 events per tick, one instance can process 200, and the topic has six partitions. Instances are varied from one to eight, and the numbers are what the model produces — a backlog in events after ten ticks, with no wait time anywhere, because a backlog count is a conservation fact and a wait time is a queueing model this domain does not own.
Read the last three rows together. At six instances the group exactly matches arrivals and the backlog is zero. At seven and eight instances the effective capacity is still 1200, the backlog is still zero, and one and then two instances are assigned nothing. The scale-up changed the cost and changed nothing else.
The final row is the fix. Raising partitions to twelve lifts the ceiling, so eight instances can now contribute 1600 of capacity against 1200 of arrivals and the group drains with headroom. That is the real lever, and it is the one with an ordering consequence attached — which is why it belongs in a design conversation rather than in an incident (Event Keys and Partition Assignment).
In the nine-on-six case, a third of the group is provisioned and assigned nothing. Pure waste, and invisible in any metric that averages utilisation across the group.
Instances on quiet partitions while one is saturated. Same waste as above at a finer grain, and it is not fixed by more instances or more partitions — only by a different key (Event Keys and Partition Assignment).
Every membership change pauses consumption while assignments settle. An autoscaler reacting to lag pays this repeatedly and can make lag worse.
The price of group independence. Real, predictable, and worth paying — it is what buys isolated replay and fan-out.
The work you actually wanted. In a badly-shaped group it is a minority of the spend, which is the point of putting it last.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a deliberately badly-shaped group, to establish an ordering rather than a magnitude. Not measurements. The teaching is that the top two lines are both "capacity that cannot be assigned work", and both are fixed by partitioning and keys rather than by scaling.
| Instances | Partitions | Effective capacity | Idle instances | Backlog after 10 ticks | Draining? |
|---|---|---|---|---|---|
| 1 | 6 | 200 | 0 | 10 000 | no — arrivals exceed capacity |
| 2 | 6 | 400 | 0 | 8 000 | no |
| 3 | 6 | 600 | 0 | 6 000 | no |
| 4 | 6 | 800 | 0 | 4 000 | no |
| 5 | 6 | 1 000 | 0 | 2 000 | no |
| 6 | 6 | 1 200 | 0 | 0 | yes — capacity exactly matches arrivals |
| 7 | 6 | 1 200 | 1 | 0 | yes, and the seventh instance contributed nothing |
| 8 | 6 | 1 200 | 2 | 0 | yes, and two instances contributed nothing |
| 8 | 12 | 1 600 | 0 | 0 | yes — raising partitions raised the ceiling |
The failures that keep the group green
Consumer groups fail in ways that leave the broker's dashboard reassuring, which is why the debugging sequence matters more than any single metric. The order that works: is maximum per-partition lag high, or only total lag? Are any instances holding zero partitions? Is one partition carrying most of the records? Is the rebalance rate elevated? Is the group committing offsets while producing no output?
Each of those questions eliminates a whole class of cause, and each of them is unanswerable from the aggregate numbers most dashboards show. A summed lag across twelve partitions is dominated by normal traffic and a single stuck partition contributes almost nothing to it.
The last row of the table is the most dangerous and the least discussed. Lag can be zero while nothing useful is happening, because a committed offset records that a consumer moved past a record, not that it did anything with it. Progress and production are two separate measurements and only one of them is the broker's to report (The Pipeline Succeeded. The Data Is Wrong.).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Lag grows; instances are added; nothing changes. | Total lag climbing steadily while CPU across the group is low and every instance reports healthy. | Instance count has passed the partition count. Effective capacity is capped at partitions × per-instance capacity and the extra members hold no assignments. | Check assigned partitions per instance. Cap the autoscaler at the partition count, then either raise partitions — accepting the key re-map — or make each instance faster (Event Keys and Partition Assignment). |
| Lag grows on one partition only; the rest are current. | Maximum partition lag is enormous, mean lag looks fine, one instance is pinned and the rest are idle. | Key skew. One key or a small set of keys carries most of the traffic and hashes to a single partition, which one instance owns. | Identify the top keys by volume. Either size for the hot partition, route that key to its own topic, or salt the key and re-aggregate downstream — the last only if per-key ordering was not the point (Salting a Skewed Key, Data Skew). |
| A deploy or a scale event, followed by a lag spike that outlasts it. | Consumption pauses across the group repeatedly; lag ratchets up with each event instead of recovering. | Rebalance churn. Membership changes faster than assignments settle, and consumption stops for the affected partitions each time. | Damp the autoscaler, lengthen its cooldown, cap it at the partition count, and prefer rolling deploys that change membership once rather than repeatedly. |
| One partition's lag rises without bound; the instance is healthy and heartbeating. | A single partition never advances. Aggregate metrics show a mildly elevated backlog. | A poison record being retried forever. There is no per-message state in a log, so nothing moves it aside on its own. | Catch the failure, publish the record to a quarantine topic, commit past it, and alert on quarantine arrivals as a data-loss signal rather than an application error (A Dead-Letter Queue Is a Workflow, Not a Bin). |
| Lag is zero and a downstream table has stopped growing. | Every broker metric is perfect. The warehouse table's row count is flat and its freshness is degrading. | The consumer is committing offsets and failing to produce output — a swallowed write exception, a misconfigured destination, a filter that now matches nothing. | Monitor output row count per interval alongside lag. Lag proves the consumer moved; only the output proves it did something (Volume Anomalies, Data Observability). |
How to build it
Most important first.
- Size partitions for the peak consumer parallelism you will ever want, because that number is the ceiling and raising it later re-maps keys (Event Keys and Partition Assignment).
- Give every distinct purpose its own group id, derived from the deployment name rather than from a constant. Two purposes sharing a group is one of the few failures here that produces missing data rather than late data.
- Cap the autoscaler at the partition count. An autoscaler that can exceed it will, and the extra instances are cost with no throughput and additional rebalance churn.
- Fix the ceiling before optimising the consumer. If arrivals exceed
partitions × per-instance capacity, no amount of tuning inside an instance closes the gap — the shape of the problem is a throughput deficit and waiting will not clear it. - Checkpoint any per-partition state so a rebalance resumes rather than restarts, and make the sink idempotent so the redelivery around a rebalance is harmless (Checkpointing, Idempotent Data Pipelines).
- Alert on maximum per-partition lag, never on the sum. A single blocked partition contributes almost nothing to a total and is the failure you most need to see (The Backlog Arithmetic: Four Levers and a Drain Time).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Each partition is consumed by at most one instance within a group at a time. This is the assignment guarantee and it is what makes per-partition ordering usable by a parallel consumer.
- Every group receives every record of the topic, independently of every other group, from its own committed position.
- Delivery within a group is at-least-once by default: after a crash or a rebalance, processing resumes from the last committed offset, so anything processed-but-not-committed is processed again (At-Least-Once Delivery).
- No guarantee of even work distribution. Assignment balances *partitions*, and partitions are not equal (Data Skew).
- No guarantee that an instance holds a partition for any length of time. Assignments are revocable at any rebalance, which is what makes in-memory state a liability.
- No guarantee of progress. An instance that is alive, heartbeating and stuck retrying one record holds its partitions and consumes nothing, and the group looks healthy.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check maximum per-partition lag against a stated freshness commitment. It catches a stalled instance, a poison record, a hot partition and a group that is simply under-provisioned (The Freshness SLO).
- It misses a consumer that is committing offsets and doing nothing useful — lag looks perfect while records are being silently skipped or written nowhere. Pair it with an output-side row count so progress and production are checked separately.
- It also misses a second group that should exist and does not. If a topic's records are supposed to reach two destinations and one group was never deployed, every lag metric on the group that does exist will be excellent (Reconciliation).
- Freshness for a group is
lag ÷ throughput, and both terms are per-partition. Averaging either across a topic produces a number that describes no partition and hides the worst one. - A group that is draining will catch up completely — the log kept the records — so a lag spike is a freshness problem rather than a data-loss problem, right up until the oldest retained record starts to age out (Retention and Replay).
- Adding instances improves freshness only while instances are below the partition count and the load is even. Past either condition the freshness curve is flat, which is why the "scale it" reflex fails so visibly here.
- Changing a group id creates a new group, which starts from whatever reset policy is configured — earliest, meaning a full reprocess, or latest, meaning a silent gap. A rename in a deployment file is therefore either a replay or data loss (Replay from the Log).
- A partition count increase changes assignments for every member, and increases the ceiling. It also re-maps keys, so it is never only a consumer-capacity change (Event Keys and Partition Assignment).
- Adding a topic to a group's subscription redistributes all partitions across both topics, so a group's throughput on the original topic changes because of a change to an unrelated one.
- Reset the group's committed offsets to an earlier position and reprocess. This is bounded by retention and safe only if the sink tolerates re-writing (Replay from the Log, Upserts and Merges).
- To reprocess without disturbing production, deploy a second group against the same topic writing to a scratch destination, validate, then swap. The production group is unaffected because groups are independent — this is the log's best operational property (Atomic Publish).
- A stuck partition is unblocked by publishing the offending record to a quarantine topic and committing past it. That is a deliberate, recordable data-loss decision rather than a routine restart (A Dead-Letter Queue Is a Workflow, Not a Bin).
- After a rebalance, correctness of any windowed aggregation depends on state having been checkpointed. Without checkpoints the recovery is a replay of the affected window, which requires knowing which window was affected (Checkpointing).
What can go wrong
- Instances beyond the partition count, idle and invisible — the failure this lesson exists for, because everything about it looks like a successful scale-up.
- A rebalance storm: instances joining and leaving faster than assignments settle, so the group spends most of its time not consuming and lag climbs during what is nominally a scaling event.
- A hot partition setting the group's throughput while aggregate utilisation looks low, and every scaling response making it worse by adding rebalances (Data Skew).
- A poison record on one partition, retried indefinitely by a healthy, heartbeating instance. Group-level metrics stay green.
- Two purposes sharing a group id, so each sees a subset of records and both destinations are quietly incomplete.
- Offsets committed before processing as a "performance improvement", turning every rebalance and every crash into silent record loss (Offsets and Commits).
- "Lag is high, so add consumers." Only helps below the partition count and only when load is even. Past the ceiling the extra instances idle, and the deployment looks entirely successful (More Threads Is Not More Speed).
- "Lag in records tells us how far behind we are in time." It tells you how many records are unread. Converting that to minutes needs a throughput assumption that is least trustworthy exactly when the backlog is large (Little's Law as Working Intuition).
- "The group is healthy, so the data is arriving." An instance can commit offsets and produce nothing. Progress and production are separate measurements and only one of them is on the broker's dashboard (The Pipeline Succeeded. The Data Is Wrong.).
- "Rebalancing balances load." It balances *partitions*. A group with one partition carrying most of the traffic is perfectly balanced by partition count and completely unbalanced by work (Data Skew).
- "Autoscaling on lag is the obvious control loop." Scaling triggers rebalances, rebalances pause consumption, and pausing consumption raises lag. Cap it at the partition count and damp it, or it will oscillate.
Operating it
- Maximum per-partition lag per group, and the partition it belongs to. The maximum is the alertable number; the sum is a number that hides the incident (The Backlog Arithmetic: Four Levers and a Drain Time).
- Assigned partition count per instance, which makes idle instances beyond the ceiling immediately obvious and is almost never on a dashboard.
- Rebalance rate per group. A rising rate is a throughput problem before it is anything else, and it is usually caused by whatever is trying to fix the throughput problem.
- Records processed per instance, next to assigned partitions per instance. Uneven records with even partitions is skew; even records with uneven partitions is an assignment problem (Six Queue Signals, Two That Wake You Up).
- Oldest retained record age against maximum lag, which is the only pair that tells you a freshness incident is about to become a data-loss incident.
- At 10x arrivals the ceiling is usually what binds first: the partition count that was generous at the old rate becomes the cap, and raising it has ordering consequences (Event Keys and Partition Assignment).
- At 100x, per-instance efficiency starts to matter as much as instance count, because the ceiling limits how much of the problem can be solved by adding machines.
- Group count scales read bandwidth and coordination linearly, and each group is another party depending on the topic's schema (Data Contracts).
- Below the ceiling and with even keys, this is one of the few places in data engineering where scaling genuinely is linear — which is precisely why the ceiling surprises people when they reach it.
- Instances beyond the partition count are pure cost: provisioned, scheduled, monitored, and assigned nothing. This is the most common wasted spend in a streaming platform and it is invisible in every utilisation metric that averages across a group.
- Skew wastes the same way at a finer grain: instances on quiet partitions are paid for and idle while the group is bottlenecked on one (Data Skew).
- Rebalances cost throughput directly — consumption pauses while assignments settle — so a churning group is paying for capacity it cannot use.
- Each additional group multiplies read bandwidth on the topic, which is the price of the independence that makes groups valuable (Kafka as a Log, Not a Queue).
- Partition-based assignment buys per-partition ordering for a parallel consumer and costs a hard, pre-committed ceiling on parallelism.
- Independent groups buy fan-out and isolated replay and cost read bandwidth multiplied by group count.
- Checkpointing state buys rebalance safety and costs write throughput and complexity in the consumer; skipping it buys simplicity and costs correctness on every rebalance, which is a routine event rather than a rare one.
Consumer lag lab
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- BROKER-SPECIFICKafka assigns whole partitions to group members with a broker-side coordinator and revocable assignments. Kinesis uses a client library that leases shards through an external table, so the failure modes are lease expiry rather than rebalance storms. Pulsar offers a shared subscription mode that dispatches individual messages and therefore has no partition ceiling at all, at the cost of per-subscription ordering.
- SIMULATEDThe capacity table below is computed by
simulateLag()insrc/de/sim/stream.ts, a conservation model over arrivals and capacity. Its numbers are unitless events per tick from a teaching model, not measurements of any cluster, and it deliberately reports no wait time because Observability & Performance owns queueing latency. - GENERALThe underlying rule — a work-partitioning scheme where a partition has at most one owner caps parallelism at the partition count — applies identically to sharded batch compute and to any leased-resource worker pool. What is broker-specific is only how ownership is granted and revoked.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the group membership problem underneath this: failure detection by heartbeat, the coordinator that decides assignments, and why a member that is slow is indistinguishable from a member that is dead. Rebalance behaviour is that theory made operational.
- — Distributed Systems also owns why revocable leases on partitions are the honest design — an assignment cannot be permanent in a system where a member can disappear without saying so — and what that implies for any state a member holds.
- — DevOps / Production Engineering owns the autoscaling control loop itself: why scaling on a lagging indicator oscillates, how to damp it, and why the cap in this lesson belongs in the deployment configuration rather than in a runbook.