Message Brokers: Log-Shaped and Queue-Shaped
Two different products wearing one word. A queue distributes work and forgets; a log stores records and lets anyone re-read them. Neither is the upgrade of the other.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Does this data need a durable replayable log, or a queue that hands each message to exactly one worker and deletes it?
On the queue side, a worker that must do a job once: send the email, charge the card, resize the image. On the log side, an unknown and growing set of readers — a stream processor, a warehouse loader, a search indexer, a team that will exist next year — none of which should affect the others. Choosing the wrong shape means one of these two groups is permanently badly served.
A queue's grain is one unit of work, and its lifecycle is claim → process → acknowledge → disappear. A log's grain is one record at an offset, and its lifecycle is append → available until retention expires. The word "message" covers both and hides the only difference that matters.
Pick whichever broker the team already runs and use it for everything. This is more defensible than it sounds — an organisation that operates one messaging system well is usually better off than one that operates three badly, and the shapes overlap enough that either can be forced into the other's job for a while.
A queue was chosen, and now analytics wants the same events. The queue has already deleted them on acknowledgement, so the only options are to have the worker re-publish everything (a second integration, with its own gaps) or to start collecting from today and accept that history does not exist (Keeping Raw History: The Recovery Position and the Liability).
- A queue was chosen, and now analytics wants the same events. The queue has already deleted them on acknowledgement, so the only options are to have the worker re-publish everything (a second integration, with its own gaps) or to start collecting from today and accept that history does not exist (Keeping Raw History: The Recovery Position and the Liability).
- A log was chosen for a work queue. Every job is now assigned to a partition, one slow job blocks every job behind it in that partition, and the natural fix in a queue — acknowledge out of order, retry that one message — is structurally unavailable (Queue Semantics).
- A dead-letter mechanism is needed. In a queue this is a built-in per-message facility. In a log it must be built: a consumer that catches, publishes to a separate topic, and commits past the bad record — and if it does not, that record stops the partition forever (A Dead-Letter Queue Is a Workflow, Not a Bin).
- The team enables competing consumers on a log-shaped broker to increase throughput and discovers that adding instances past the partition count does nothing, because a partition is assigned to at most one consumer in a group (Consumer Groups and the Parallelism Ceiling).
- Priority is required — expedited orders before standard. Queues can be given priority semantics; a log is a sequence and has no notion of jumping the line, so the only implementation is a second topic and a consumer that reads it preferentially.
What is actually happening
- A queue-shaped broker tracks per-message state. A message is invisible while a consumer holds it, redelivered if the acknowledgement does not come in time, moved aside after a configured number of failures, and deleted on success. That per-message bookkeeping is the product, and it is why queues can retry, dead-letter and prioritise individual messages.
- A log-shaped broker tracks nothing per message. It appends bytes and stores integers — one committed offset per consumer group per partition. That is why it scales to enormous fan-out and why it cannot express "this one message failed, keep the rest moving": there is no per-message state to put that in.
- Everything else falls out of those two data structures. Replay exists in a log because records survive being read. Selective retry exists in a queue because the broker knows which message you have. Ordering is per-partition in a log because a partition is a file; ordering is best-effort in most queues because messages are handed out independently (Topics and Partitions).
- The shapes converge in the middle and it is worth being honest about it. Several queue products can retain acknowledged messages for a window; several log products layer per-message acknowledgement and dead-letter behaviour on top of the log. The convergence is real and it is still not symmetric — the underlying structure decides which behaviours are cheap and which are bolted on.
- For data engineering the tiebreaker is almost always fan-out plus replay. A data platform's consumer set grows and its transformations get bugs, and both of those are log properties (Replay from the Log).
The same word for two different data structures
Comparisons of brokers usually turn into feature lists, which is why they do not transfer. The useful comparison is structural: one product keeps state per message and deletes on success, the other keeps an offset per consumer group and deletes on a timer. Every row in the table below is a consequence of that, not an independent design choice.
Read the table looking for which properties you actually need rather than which column has more of them. Most teams need per-message failure handling for operational work and replay for analytical work, which is why the honest answer is frequently "both, with a clear boundary".
Notice what is identical in both columns: delivery is at-least-once, completeness is not guaranteed, and acknowledgement proves nothing about correctness. The things brokers do not do for you do not vary by shape.
| Property | Log-shaped (Kafka, Pulsar, Kinesis) | Queue-shaped (SQS, RabbitMQ) |
|---|---|---|
| What the broker stores per message | Nothing. Bytes in a partition plus one committed offset per consumer group. | Per-message state: visible / in-flight / acknowledged, attempt count, visibility deadline. |
| Effect of a read | None. The record stays until retention expires. | The message becomes invisible to others, then is deleted on acknowledgement. |
| Second independent consumer | Trivial: a new consumer group reads the same records from any retained offset. | Requires a fan-out mechanism that copies each message into a second queue, arranged in advance. |
| Replay of last week | Seek to an earlier offset or timestamp. This is the feature the product exists for. | Not possible. Acknowledged messages are gone. |
| Ordering | Total order within a partition; none across partitions. | Best-effort in standard modes; strict FIFO available in a dedicated mode, usually at a throughput cost. |
| One bad message | Blocks its partition unless the consumer explicitly catches, side-publishes and commits past it. | Redelivered a bounded number of times, then dead-lettered. The rest of the queue keeps moving. |
| Consumer scaling | Capped by partition count within a group; instances beyond it are idle. | Add workers freely; they compete for the same messages. |
| Priority / expedite | No native concept. Implemented as a separate topic. | Native in several products, or approximated with separate queues. |
| Dominant cost driver | Bytes retained × replication factor, plus bytes read × consumer count, plus cross-zone traffic. | Message and operation counts, multiplied by every retry. |
| Delivery semantics | At-least-once. Effectively-once effects require an idempotent or transactional sink. | At-least-once. Effectively-once effects require an idempotent or transactional sink. |
| Completeness relative to the source | Not guaranteed. Reconciliation is the only detector. | Not guaranteed. Reconciliation is the only detector. |
Product capabilities move across this boundary over time — retention windows on queue products, subscription modes on log products, tiered storage that changes the retention cost curve. Treat the structural column headings as stable and verify any specific product behaviour against current documentation before designing around it.
Choosing, in terms of what you will need on a bad day
The choice is easiest to make from the failure end rather than the feature end. Two questions decide it almost every time: when one item cannot be processed, must the rest keep moving? And when you discover a bug six weeks from now, will you need the original records back?
A "yes" to the first pulls hard toward a queue, because per-message failure handling is structural there and manual everywhere else. A "yes" to the second pulls hard toward a log, because there is no way to reconstruct records a queue deleted. If both are yes — which is common — the answer is a log as the durable record and a queue for the work derived from it, and the boundary between them is a consumer you write.
What should not decide it: which product is newer, which has more adoption, or which the team finds more interesting. Those arguments produce a work queue with partitions or an analytics platform with no history, and both take a long time to undo.
What does this data need on the day something goes wrong?
when Multiple independent consumers now or plausibly later; history matters; reprocessing after a logic bug is a requirement; per-key ordering matters.
cost Schema compatibility becomes a hard constraint across the whole retention window; partition count must be planned because changing it re-maps keys; per-message retry must be built by hand.
when One logical worker per message; per-message retry and dead-lettering matter; jobs are independent; nobody will ever need to re-read yesterday.
cost No replay, no history, and no second consumer without a fan-out copy arranged in advance. If analytics wants this data later, it starts from today.
when The events have analytical value and some of them trigger operational work that needs isolated retry — the common shape for a platform of any size.
cost Two systems, two sets of failure modes, and a hand-off consumer that becomes a place data can be lost if it is not itself monitored and idempotent.
when One producer, one consumer, inside one deployment, at a volume a database table comfortably handles.
cost None, and this is a legitimate answer. Keep the events in a table you never delete from, so the history exists when you do need a broker (Keeping Raw History: The Recovery Position and the Liability).
Where the broker sits in a data platform
For a data platform the broker is rarely the destination. It is a boundary — the place where the producing system's obligations end and yours begin — and its most valuable property is that it decouples the producer's deployment schedule from every consumer's.
The stage table below is worth reading down the guarantees column. The broker's promise is narrow and precise, and the two stages either side of it are where completeness and correctness are actually established. Teams that treat the broker as the guarantee are the ones surprised by missing rows that the broker was never in a position to prevent.
The pattern worth adopting from this is that the first consumer of a topic in a data platform should usually be a raw landing writer that does nothing but persist what arrived, exactly as it arrived. It converts the broker's finite retention into your indefinite history, and it is the cheapest insurance in the module (The Raw Landing Zone).
- 1Producing service
Commits its business transaction and publishes an event describing what happened.
guarantees That the event describes a committed change — and only if publish and commit are tied together, which by default they are not.
fails by Committing the database write and failing the publish, producing a change no downstream will ever see (The Transactional Outbox).
- 2Broker
Accepts the append, replicates it, retains it, serves it to any number of readers.
guarantees Durability at the requested replication level, immutability, ordering within a partition, availability until retention expires.
fails by Retention expiring under a lagging consumer; a partition count change silently re-mapping keys (Event Keys and Partition Assignment).
- 3Raw landing writer
Reads the topic and writes records untouched to object storage, partitioned by arrival.
guarantees That the broker's finite history becomes durable history you control, in the form it arrived.
fails by Being "helpful" — parsing, filtering or reshaping on the way in, which destroys the only artefact that could prove what the producer actually sent.
- 4Deduplication and ordering
Collapses redeliveries on the event id and reconstructs per-entity ordering from event time or source sequence.
guarantees One row per real-world occurrence, at a declared grain, given a stable identifier exists.
fails by Deduplicating on a window too short for the real redelivery gap, or ordering by arrival when the producer's order is what matters (CDC Ordering and Transaction Boundaries).
- 5Transformation and model
Turns records into facts and dimensions at a declared grain.
guarantees Only what its tests assert. By default nothing (Data Tests).
fails by Counting records as entities — three change records for one order are three rows and one order (Grain: What Does One Row Represent?).
- 6Serving table
Holds the modelled result for consumers to query.
guarantees Atomic visibility of a published version, if it was built that way (Atomic Publish).
fails by A replay overwriting it while consumers read, so a report captures a state that never existed as a whole.
The broker's row is the narrowest promise in the table, and the two rows around it are where the real work is. A platform that assumes the broker guarantees completeness has no stage that establishes it.
How to build it
Most important first.
- Ask what happens on the day a second consumer wants the same data. If the answer must be "it just reads it", you need a log, and choosing anything else now is a migration later.
- Ask whether failure is per-message or per-stream. A single poisonous record that must not block the others is a queue requirement; a stream where record N+1 is meaningless without record N is a log requirement.
- Use both where both apply, and be explicit about the boundary. A very common and very sound architecture is a log as the platform's durable spine, with a consumer that turns selected records into jobs on a queue for work that needs per-message retry (Job Queues).
- Never make the data platform a consumer of a work queue. Analytics reading from the same queue that drives operational work couples two teams' failure domains and gives the platform no replay (Data Engineering and Backend Engineering).
- Whichever shape you pick, put deduplication downstream. Both shapes are at-least-once in practice, and any design whose correctness depends on a broker not redelivering is already broken (Deduplication).
- Decide the retention or the redelivery limit deliberately and write down the consequence. Both are the point at which data stops being recoverable and neither has a safe default (Retention and Replay).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Queue-shaped: at-least-once delivery to one consumer, per-message acknowledgement, automatic redelivery on timeout, a bounded number of attempts before dead-lettering. Ordering is typically not guaranteed unless a FIFO mode is explicitly selected, and that mode usually costs throughput.
- Log-shaped: durable retention independent of consumption, ordering within a partition, independent positions per consumer group, and replay to any retained offset. No per-message acknowledgement and no built-in notion of a failed message.
- Neither guarantees exactly-once delivery. Both can be made to produce effectively-once *effects* if the consumer's write is idempotent or transactional, and that is a property of your sink, not of the broker (Offsets and Commits).
- Neither guarantees completeness relative to the producing system. A message that was never published is not a broker failure and no broker metric will show it (Reconciliation).
- Neither guarantees that a consumer that acknowledged actually succeeded. Acknowledgement means the consumer said so.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- On a queue, monitor dead-letter arrivals as a data-loss signal, not as an application error signal. A message in a dead-letter queue is a record your platform does not have, and the check that catches it is a count of DLQ arrivals per period compared with zero.
- On a log, monitor the gap between the oldest retained record and the furthest-behind consumer. That is the equivalent measure of impending loss.
- Both checks miss the same thing: a message that was never produced. Only reconciliation with the source system closes that gap, and it must be run against a closed period so that in-flight messages are not counted as losses (Reconciliation).
- Both remove polling from the freshness budget and neither makes a consumer faster. Where they differ is in what happens when the consumer cannot keep up: a queue grows a backlog of undelivered messages and may hit a queue depth limit, while a log grows consumer lag and hits nothing at all until retention.
- That difference matters for a data platform. A log lets a consumer be hours behind and catch up completely; a queue with a depth limit or a message TTL turns a slow consumer into permanent loss (The Backlog Arithmetic: Four Levers and a Drain Time).
- Queue-shaped brokers can offer lower per-message delivery latency for small volumes because there is no batching to wait for. Log-shaped brokers batch to get throughput, and the batch interval is a freshness-versus-cost knob you own (Cost vs Freshness).
- On a queue, a schema change affects only messages in flight — the backlog drains within minutes or hours and the new shape takes over. A short-lived, forgiving situation.
- On a log, a schema change affects every retained record and every future replay. Consumers must read both shapes for as long as retention lasts, which makes compatibility a hard constraint rather than a deployment-ordering problem (Backward Compatibility).
- This asymmetry is a real and rarely-stated cost of the log: it makes evolution slower and more disciplined, permanently, in exchange for the replay you bought it for (Schema Evolution).
- A queue's recovery story is redelivery and the dead-letter queue: failed messages come back, up to a limit, and then go somewhere a human can look. Reprocessing yesterday is not a feature — the messages are gone.
- A log's recovery story is offset reset: point a consumer group at an earlier position and reprocess. Recovery is bounded by retention, not by whether anything failed (Replay from the Log).
- Replaying a dead-letter queue back into the main queue is a legitimate and underused recovery move, but it re-delivers in a different order than the original, so any consumer that depends on ordering will produce a different result than it would have (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Either way, the sink must be idempotent before recovery is safe. Redelivery and replay both write records the sink has already seen (Idempotent Data Pipelines).
What can go wrong
- A queue with an unbounded backlog and no alerting, so a stalled consumer is discovered when the broker hits a storage limit and starts rejecting producers (Bounded vs Unbounded Queues).
- A poison message in a log-shaped broker, retried forever by a consumer that cannot skip it, blocking every subsequent record in that partition while the other partitions look healthy.
- A dead-letter queue nobody reads. It is a data-loss bucket with a reassuring name, and in most organisations it is full.
- A team using a log as a work queue and adding consumers to increase throughput, with no effect, because the partition count caps parallelism (Consumer Groups and the Parallelism Ceiling).
- A queue with a message TTL, chosen as a hygiene measure, silently discarding messages during an incident — the exact moment the backlog is large and the data matters most.
- A retry policy on the queue interacting with a retry policy in the consumer, so one failing dependency produces a multiplied load that keeps it failing (Retry Storms: The Load You Generated Yourself).
- "Kafka is a message queue." It is a log, and the difference is not terminology — a queue deletes on consume and a log does not, which is why one supports replay and independent consumers and the other supports per-message retry (Kafka as a Log, Not a Queue).
- "The log is the modern choice, queues are legacy." Both are current and both are correct for different jobs. Sending one email and no more than one is a queue problem, and no amount of replay capability makes it a log problem (Job Queues).
- "We can add replay to our queue later." Replay requires that the records were kept. A queue that has already acknowledged and deleted them cannot be retrofitted with history that no longer exists.
- "More consumers means more throughput." On a queue, usually. On a log, only up to the partition count, and the instances beyond it are idle (Consumer Groups and the Parallelism Ceiling).
- "Dead-letter queues solve poison messages." They quarantine them. Whether they solve anything depends entirely on whether a human process reads that queue, and the honest default assumption is that nobody does.
Operating it
- Queue: depth, oldest-message age, in-flight count, redelivery rate, dead-letter arrival rate. Oldest-message age is more useful than depth because it is comparable against a commitment (Depth Is Not an Emergency; Age Is).
- Log: per-partition consumer lag in records, oldest retained record age, and offset-reset events. Lag in records is the number to alert on; converting it to a time estimate requires a throughput assumption that is wrong exactly when it matters (The Backlog Arithmetic: Four Levers and a Drain Time).
- Both: producer-side error and retry rate, which is the only signal for messages that never made it in (Six Queue Signals, Two That Wake You Up).
- Both: consumer error rate broken down by whether the record was eventually processed, dead-lettered or skipped. "Errors" without that breakdown cannot distinguish a transient blip from silent loss.
- A queue scales workers freely — add consumers and each takes messages from the same pool. That is a genuine advantage over the log at 10x work volume and it disappears the moment ordering matters.
- A log scales readers freely and caps writers per key and consumers per group at the partition count. At 10x you add partitions; at 100x you discover that adding partitions is not free because it re-maps keys (Event Keys and Partition Assignment).
- At high consumer counts the log wins decisively: the tenth consumer of a topic costs read bandwidth, while the tenth independent consumer of a queue needs its own copy of every message via a fan-out mechanism.
- At very high message rates with tiny messages, the queue's per-message bookkeeping becomes the bottleneck long before the log's sequential append does (Throughput: Requests, Packets and Bytes per Second).
- Queue-shaped: cost tracks the number of messages and the number of operations against them — every delivery, every acknowledgement, every redelivery is work the broker does per message. Retries are therefore a direct cost multiplier.
- Log-shaped: cost tracks bytes appended, bytes retained (times replication factor), bytes read (times consumer count), and cross-zone traffic. Per-message bookkeeping is absent, which is why fan-out is comparatively cheap.
- The log's retained-bytes line grows with the recovery window you chose and stays there. It is the only cost in this module that accumulates whether or not anyone is using the system (Storage Lifecycle).
- Operating cost is the line usually left out and often the largest: a self-managed log-shaped broker is a distributed storage system with partitions, replicas, rebalances and disk pressure (Managed vs Self-Hosted).
- Choosing a log buys replay, fan-out and history and costs schema discipline, partition planning, and the loss of per-message retry semantics you will miss the first time one record is malformed.
- Choosing a queue buys per-message failure handling, trivial worker scaling and operational simplicity, and costs any possibility of a second consumer or a reprocess after the fact.
- Running both is the honest answer for most platforms and costs two systems to operate, two sets of failure modes to learn, and a boundary between them that has to be explicit or it becomes a place data goes missing.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe structural split — per-message state with destructive read, versus offset-addressed retention with non-destructive read — is what decides which behaviours are cheap. It holds regardless of product and predates all of the current ones.
- BROKER-SPECIFICKafka and Kinesis are log-shaped with no per-message acknowledgement; SQS and RabbitMQ are queue-shaped with per-message acknowledgement and built-in dead-lettering; Pulsar is log-shaped storage exposing queue-like subscription modes, so it genuinely sits between the two and should not be filed under either.
- ORG-SPECIFICA team that already operates one broker competently will often be right to stretch it into the other shape rather than take on a second system. That trade is about operational capacity, not architecture, and it should be argued in those terms rather than dressed up as a technical preference.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the delivery-semantics taxonomy itself — at-most-once, at-least-once and what "effectively-once" can mean across a network — and the reason no broker on either side of this comparison can offer a stronger one on its own.
- — DevOps / Production Engineering owns operating these brokers: capacity, rebalancing, upgrades and the runbook for a partition without a leader. This lesson chooses a shape; that domain keeps it alive.