The question this answers
Do I need a queue or a topic — and why does the answer keep being "a log"?
A work queue guarantees each message is processed by one consumer of the pool (at least once under failure). A pub/sub topic guarantees each *subscription* receives its own copy (at least once per subscription). A log with consumer groups guarantees both: exclusive partition assignment inside a group, independent positions across groups.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A consumer knows only which messages it was given. In a work queue it cannot tell whether a message went to a peer or was never published; in pub/sub it cannot tell whether a sibling subscription saw the same message. The choice of model determines what is knowable, and no amount of instrumentation inside one consumer recovers what the model did not preserve.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The rule, stated plainly
If exactly one worker should do this task, use a queue. If several independent parties need to know this fact, use pub/sub. That is the whole rule, and it resolves the great majority of real decisions on the first pass.
The reliable tell is grammatical. A queue carries commands: ResizeImage, SendInvoice, ChargeCard — imperative, one correct handler, failure means the work did not happen. A topic carries events: OrderPlaced, UserSignedUp, PaymentSettled — past tense, no implied handler, and whether anyone acted is not the publisher’s concern.
Getting it backwards produces two distinctive bugs. Commands on a topic mean N subscribers each do the work — N charges, N emails. Events on a queue mean only one of your interested consumers sees each event, so every consumer appears to be missing a random fraction of the data. The second is much harder to spot, because the system looks like it is losing messages rather than misrouting them.
| Work queue | Pub/sub topic | Log + consumer groups | |
|---|---|---|---|
| Who processes a messageprotocol | One consumer of the pool | One consumer per subscription | One consumer per partition per group |
| Read is destructiveprotocol | Yes — ack deletes | Yes, per subscription copy | No — position advances, data stays |
| Replay after processingprotocol | Impossible | Impossible | Reset the offset |
| Add a consumer later, with historyprotocol | No | No | Yes, within retention |
| Orderingtypical | None with >1 worker | Per-subscription at best | Total within a partition |
| Parallelism ceilingprotocol | Unbounded — add workers | Unbounded per subscription | Partition count per group |
| Per-message redelivery of one failuretypical | Natural — that one message returns | Natural per subscription | Awkward — the position is per partition, not per message |
| Operational weighttypical | Low | Low to medium | High |
Where the rule bends: one fact, several handlers, and retry granularity
The first real complication is that the same happening is often *both*. An order being placed is a fact (analytics and fraud want it) and also triggers a command (charge the card). The correct shape is usually to publish the fact once and let a subscriber issue the command — event to topic, command to queue — rather than trying to make one channel do both jobs.
The second, less obvious complication is retry granularity, and it is the strongest surviving argument for classic queues. In a queue, one message failing affects one message: it is redelivered on its own while everything else flows. In a log, position is per partition, so a message that will not process is sitting at the head of an ordered stream. You must either stop the partition (head-of-line blocking for every key that hashes there) or skip past it and handle the failure out of band. There is no third option, because the ordering guarantee and per-message retry are fundamentally in tension.
This is why mature log-based pipelines end up re-implementing a queue at the edge: failures are diverted to a retry topic or a DLQ so the main partition keeps moving. You did not escape the queue; you bounded where it is allowed to exist.
- Command → queue: one handler, failure is meaningful to the sender, per-message retry matters.
- Event → topic or log: many handlers, the publisher does not care, ordering per key often matters.
- Need both fan-out *and* per-key ordering *and* replay → log with consumer groups, and accept the operational weight.
- Need per-message retry with fine granularity and no ordering requirement → a work queue is still the better tool, and choosing it is not a step backwards.
Why the log displaced both
A A Topic Is Not One Log: Ordering Lives Inside a Partition with Consumer Groups: Queue Semantics Inside, Pub/Sub Semantics Across subsumes the two models rather than compromising between them. Inside a group, partitions are assigned exclusively, so each message is handled by exactly one member — that is a work queue, with per-key ordering as a bonus. Across groups, positions are independent, so every group sees every message — that is pub/sub, with one stored copy instead of N.
And because reads are non-destructive, the log adds a third property neither classic model can offer at all: a new consumer can start from the past. A team that appears in month six can replay six months of history into a fresh view without the producer knowing. That capability is what makes Materialized Views: A Read Model That Lags and Architecture’s event-sourcing practical, and it is the reason log-based brokers became the default backbone for event-driven systems.
The honest counterweight: the log is heavier to operate, parallelism is capped by partition count rather than by worker count, changing that count reshuffles key placement, and per-message retry is genuinely awkward. For a service that resizes images, a plain queue remains the correct engineering answer, and reaching for a log is a cost with no matching benefit.
A decision procedure you can actually run
Work down this list and stop at the first answer that forces your hand. The order matters: the earlier questions are about correctness, the later ones about cost.
Notice that "which broker do we already run?" is not on the list until the end — but in practice it should be, and honestly. Running a second messaging system to get a marginally better fit is a real, recurring operational cost that usually outweighs the fit.
11. Is it a command (one correct handler) or an event (a fact)?2 command -> queue. event -> continue.3 42. Will more than one independent consumer ever need it?5 no -> queue is fine, and simpler.6 yes -> continue.7 83. Do you need per-key ordering?9 yes -> log, partitioned by that key. (topics generally cannot)10 no -> continue.11 124. Will a future consumer need history it never saw live?13 yes -> log, and size retention for that use case deliberately.14 no -> continue.15 165. Is per-message retry granularity important, and ordering not?17 yes -> topic/queue beats a log here; do not fight the log.18 196. What do you already operate well?20 Prefer it unless steps 3 or 4 said otherwise. A second broker is a21 permanent tax paid by whoever is on call, not by whoever chose it.Key points
- One worker should do it → queue. Several independent parties need to know → pub/sub. That resolves most decisions immediately.
- Commands are imperative and have one correct handler; events are past-tense facts with none. The grammar is a reliable diagnostic.
- Events on a work queue look like random message loss to every consumer — the hardest version of this bug to identify.
- A log with consumer groups gives queue semantics within a group and pub/sub semantics across groups, from one stored copy, plus replay.
- The log’s real cost is per-message retry granularity: position is per partition, so one bad message blocks a partition or must be diverted out of band.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Classify the message: does it instruct, or does it report? This decides command versus event.
- • Count the independent consumers that must each see it — not the number of workers, the number of distinct interests.
- • Establish whether ordering is required, and at what scope: globally (rarely achievable), per key (achievable with partitioning), or not at all.
- • Establish whether history must be readable by a consumer that does not exist yet.
- • Pick the weakest model that satisfies all four, then check the retry-granularity cost before committing.
- • Events routed through a work queue: each consumer silently receives a fraction of the stream.
- • Commands published to a topic: every subscriber executes the side effect.
- • A log chosen for fan-out where per-message retry was the real requirement, producing chronic head-of-line blocking.
- • A queue chosen where a second consumer appears later, and the history that consumer needs no longer exists.
- • Partition count chosen for today’s throughput, capping tomorrow’s consumer parallelism.
- • Phantom message loss: two services consume the same work queue and each reports processing roughly half the events. Broker metrics show zero loss, delivery counts match publish counts exactly, and the discrepancy only appears when someone compares two consumers’ record counts.
- • Duplicated side effects from a command on a topic: three subscribers each send the invoice. The operator sees three identical emails and three ledger rows, all from "successful" handlers.
- • Partition stall in a log-based pipeline: one unprocessable message halts a partition. Lag climbs on that partition alone; aggregate consumer lag looks acceptable because the other partitions are fine, so the alert never fires.
- • Retention regret: a new team asks for history, and it was never there. The operator has to answer "we can start you from today", which is a design decision that was made silently months earlier by picking a queue.
- • Broker sprawl: three messaging systems in one estate because each decision was made locally and optimally. On-call now needs three sets of runbooks for the same class of incident.
- • A work queue coordinates through the broker as a single arbiter of claims — cheap, centralised, and bounded by that broker’s availability.
- • Pub/sub requires no coordination between subscribers, which is exactly why cross-subscriber invariants cannot exist.
- • Consumer groups coordinate membership and partition assignment, which is real distributed coordination with a real cost: Rebalancing: Everyone Stops So the Partitions Can Move is a stop-the-world event that classic queues simply do not have.
- • A queue under consumer failure redelivers individual messages, and the rest of the stream is unaffected.
- • Pub/sub under one subscriber’s failure isolates that subscriber; the others do not notice, and neither does anything else.
- • A log under consumer failure stalls or reprocesses at partition granularity, so a single bad message has a blast radius of every key in that partition.
- • Detect: for a queue, backlog age; for pub/sub, per-subscription backlog age; for a log, per-partition lag, because the aggregate hides a single stalled partition.
- • Contain: for a stalled partition, divert the offending message to a retry topic or DLQ so the partition resumes; do not let one message hold a key range hostage.
- • Recover: drain per consumer group independently — groups are isolated by design, and forcing joint recovery discards that benefit.
- • Reconcile: if you discover events were routed through a work queue by mistake, the missing data is not recoverable from the broker; you must rebuild from the source of truth.
- • Verify: confirm each consumer group and each subscription is at expected lag, not just the aggregate.
- • Per-partition lag for logs; aggregate lag is the metric that hides the incident you care about.
- • Consumer count per queue and per group, so a scale-to-zero is distinguishable from a slow consumer.
- • Duplicate-effect rate for anything command-shaped, which is the early symptom of a command on a topic.
- • Record-count parity between consumers that should see identical streams — the only cheap detector for events-on-a-queue.
- • Retention headroom: oldest retained offset versus the slowest consumer’s position, which is your remaining margin before silent loss.
- • Making the choice explicitly at design time, when changing it costs a config change rather than a data migration.
- • Choosing a log when you can already name a second consumer, or when per-key ordering is a stated requirement.
- • Choosing a queue when the workload is genuinely a pool of independent tasks — the simplicity is worth real money.
- • Adopting a log "to be future-proof" with one consumer, no ordering requirement and no replay use case. You bought partitions, rebalancing and per-partition lag alerts for nothing.
- • Forcing one model onto a system that needs both, instead of publishing the fact and queueing the command.
- • Running an extra broker to get a marginally better fit for one pipeline.
- • Publish the event to a topic or log and have a subscriber enqueue the command onto a work queue — the composition, rather than a compromise.
- • A database table as the queue, when volume is modest: same claim semantics, no dual-write problem, one fewer system.
- • Direct synchronous calls to a small, stable set of consumers, when the set genuinely does not change and the availability coupling is acceptable.
- • Change data capture off the source database instead of publishing events at all — the log comes free from the storage engine, at the cost of coupling consumers to your schema.
Queue or topic — and why the answer keeps being “a log”
| Each message goes to | Read is | Position is | History | |
|---|---|---|---|---|
| Work queuetypical | one consumer of the pool | destructive | per message (in-flight or not) | none once drained |
| Pub/sub topictypical | every subscription, one copy each | destructive per subscription | per message per subscription | usually none |
| Log + consumer groupstypical | one member per group, every group | non-destructive | an offset per partition per group | the retention window |
What people believe, and what is true
Pub/sub is just a queue with multiple consumers.
A queue with multiple consumers gives each *one* of them each message. Pub/sub gives each subscription *its own copy*. These are opposite behaviours with the same casual description.
A log is a faster queue.
It is a different data structure. Reads are non-destructive, position is per partition rather than per message, and retention is time- or size-based rather than consumption-based. Speed is not the distinction.
Kafka replaces RabbitMQ.
It replaces it for ordered, replayable, multi-consumer streams. For per-message retry, priority, and fine-grained routing to a pool of interchangeable workers, a classic broker is still a better fit.
We can decide later.
The choice determines whether history exists. A queue that has drained cannot be replayed, so "later" only ever has one available answer.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One handler should do it → queue. Everyone interested must see it → pub/sub. A log with consumer groups gives both from one copy, and adds replay.
Practical
Route commands to queues and events to topics or logs, and compose the two rather than forcing one channel to do both. If you choose a log, alert on per-partition lag and build the retry-topic escape hatch before you need it. If you choose a queue, write down what you are giving up: replay, and any future second consumer.
Advanced
The models differ in where consumption state lives. A queue stores it *in the message* (claimed, acked, deleted), so state is per message and the structure must be mutable — which forbids replay and forbids a second reader. A log stores it *in the consumer* (an offset), so the data is immutable and any number of readers can hold independent positions — which grants replay and fan-out, and costs you per-message retry, because an offset can only move past a message, never around it. Everything else in this comparison is a consequence of that one placement decision.
Apply it
- 🔧 Take an existing topic and enumerate its subscribers; classify each message type as command or event and find at least one misrouted case.
- 🔧 Implement the same fan-out twice — once with per-subscription queues, once with consumer groups over a log — and compare stored bytes and the cost of adding a fourth consumer.
- ⚡ Two consumers read the same queue and each reports half the expected records. Walk through the diagnosis.
- ⚡ A log-based pipeline has one partition stuck for two hours and aggregate lag looks normal. What alert was missing and what is the fix that does not require deleting the message?
- 💬 When would you pick a work queue over Kafka in 2026? Give me a concrete workload.
- 💬 A team put
SendInvoiceon a topic with three subscribers. What happens, and how would you have caught it in review? - 💬 What does a log give you that a topic with durable subscriptions cannot?