Kafka-Style Logs: Topics, Partitions, Offsets

A topic is a name. Its data lives in N partitions, each an append-only file on disk. A record is written to exactly one partition, chosen by hash(key) mod N — or round-robin when there is no key — and receives an offset, its position in that partition. Offsets are per partition: orders-0 has offsets 0, 1, 2, … and so does orders-1; there is no global sequence number across the topic. Brokers replicate each partition to a few others (replication factor 3; a write is acknowledged when the in-sync replicas have it, acks=all).

Nothing is deleted when a consumer reads. Retention deletes by age or size (7 days, 1 TB) regardless of who has consumed. A consumer group is a named set of consumers; each partition is assigned to exactly one member of the group, and the group commits an offset per partition (group inventory: orders-0 → 1842). Two groups reading the same topic are independent — this is how one stream feeds many teams — and a new group starting at earliest gets the whole retained history: replay.

Ordering is guaranteed only within a partition, so the key is the ordering unit. Key by orderId and every event for one order — Created, Paid, Shipped — lands in one partition in write order; consumers see a coherent story per order, and different orders interleave, which nobody cares about. Key by userId and one user’s orders are ordered relative to each other too, but a single very active user (a marketplace reseller, a load test) becomes a hot partition that one consumer must drain alone while the others idle. No key at all spreads records perfectly and maximises throughput, and destroys ordering entirely — OrderUpdated can be read seconds before OrderCreated. That exact change, orderId → random key "to fix a hot partition", is the challenge events-out-of-order.

Changing the partition count changes hash(key) mod N, so events for one key move to a different partition from that moment: old events in partition 2, new ones in partition 5, and ordering across the boundary is lost for in-flight keys. Choose the partition count with headroom up front (partitions are cheap; 12–50 is ordinary) and treat repartitioning as a migration with a drain, not a config toggle.

Consumer lag — latest offset minus committed offset, summed over partitions — is the metric that tells you whether the system is keeping up, and it should be on the first dashboard. Lag that grows is a consumer slower than its producer, exactly the situation Backpressure describes; lag on one partition only is a hot key or a stuck consumer. Express it in time too (the timestamp of the oldest unconsumed record): "1.2 M records behind" is meaningless, "38 minutes behind" is an SLO breach.

When a consumer joins or leaves a group — deploy, crash, scale-out — the group rebalances: partitions are reassigned, and during the rebalance the affected partitions are not consumed. A group of 20 consumers redeploying one at a time triggers 20 rebalances; cooperative/incremental assignment and static membership reduce the pause, but rebalances remain a deploy-time source of lag spikes. A group can never use more consumers than there are partitions; the extras sit idle.

Compaction is the alternative to time-based retention: keep only the latest record per key, delete older ones. A compacted customer-profile topic is a changelog whose replay produces the current state of every customer, the same idea as a Hash Map rebuilt from its update history, and the basis for the read models in CQRS. Commit offsets after the effect is durable, and make the effect idempotent: at-least-once with an idempotent consumer is the honest form of exactly-once, here as everywhere (Event-Driven Architecture).