AsyncCLOUD-SPECIFICSIMPLIFIEDGENERAL

Queue Semantics

Ordering, duplicates, retries, visibility timeouts and poison messages — the five properties that differ between every broker you will use.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What does my queue actually guarantee about order and delivery, and what is my code responsible for instead?

The requirement

Two events for the same order — created, then cancelled — must not be applied in the wrong order, and neither must be applied twice.

The obvious build

The queue is FIFO and delivers each message once. That is what a queue is, so the worker can assume order and assume a single delivery.

Why it breaks

FIFO describes the *queue*, not the delivery. With more than one consumer, message 1 and message 2 are handed to different workers at the same time and finish in whatever order their dependencies allow. Order of delivery is not order of completion.

How it breaks in production

FIFO describes the *queue*, not the delivery. With more than one consumer, message 1 and message 2 are handed to different workers at the same time and finish in whatever order their dependencies allow. Order of delivery is not order of completion.
A retry reorders by construction. Message 1 fails and comes back after a backoff delay; message 2 succeeded in the meantime. The cancel is applied before the create.
Duplicates are the normal case, not an exception. A worker that completed the work and died before acking will receive the message again, and the broker cannot tell that from a worker that died before starting (At-Least-Once Delivery).
A message that always throws is retried until its attempt limit, occupying a consumer each time. On a partition-ordered log it is worse: it blocks everything behind it until it is skipped (Dead-Letter Queues).
The visibility timeout that seemed generous is shorter than the p99 job duration, so the slowest jobs are being processed twice, concurrently, by design.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

At-least-once is the normal delivery guarantee. Every mainstream broker defaults to it, because the alternative — deleting the message before the work is confirmed — loses work. Plan for duplicates as the ordinary case rather than as a rare fault.
Some products offer stronger-sounding guarantees, and it is essential to separate two different claims. A broker can deduplicate *deliveries* within a window or a session — SQS FIFO deduplicates on a message id over a limited window, Pub/Sub offers an exactly-once delivery mode for pull subscriptions, Kafka offers exactly-once semantics for read-process-write cycles that stay inside Kafka. None of them makes *your* side effect happen once: your charge, your email and your third-party API call are outside every one of those boundaries.
Therefore the only construction that gives once-only business behaviour is at-least-once delivery plus an idempotent consumer. That combination is what people mean when they say exactly-once works in practice (Job Idempotency).
Ordering is a property of a scope, never of a queue as a whole. Kafka orders within a partition. SQS FIFO orders within a message group. Pub/Sub orders within an ordering key. RabbitMQ orders within a queue and only with a single consumer. In every case, the scope is what you must design around, and concurrency within that scope destroys it.
Visibility timeout (SQS), ack deadline (Pub/Sub), redelivery on channel close (RabbitMQ) and partition reassignment on poll timeout (Kafka) are four genuinely different mechanisms for the same idea. Only two of them are a per-message timer, and only one of them can be extended per message by your code.
Poison messages behave differently by shape. In a queue, a poison message occupies one consumer per attempt and then dead-letters. In a partitioned log, it halts consumption of its partition until the consumer skips or diverts it — a queue degrades, a log stops.

Five properties, five brokers, no universals

CLOUD-SPECIFICA summary of documented behaviour at the time of writing, and every cell is a setting or a mode that can change. Treat it as a map of what to go and check for your broker and version, not as a specification. §99: these products are not 1:1 equivalents, and the differences in the claim and poison-message rows are the ones that break ports between them.

This table exists because the single most expensive mistake in this module is carrying an assumption from one broker to another. A team that learned queues on SQS will assume a per-message timer; a team that learned on Kafka will assume replay is always available; both will be wrong on RabbitMQ.

Read the "poison message" column carefully. It is the property that differs most in *kind* rather than in degree: in a queue a bad message costs you one consumer per attempt, and in a log it costs you the whole partition until someone intervenes.

	SQS (standard)	SQS FIFO	Pub/Sub	RabbitMQ	Kafka	DB-backed
Delivery	At-least-once	At-least-once, deduplicated over a bounded window	At-least-once; optional exactly-once delivery mode on pull subscriptions	At-least-once with consumer acks	At-least-once per consumer; EOS only for Kafka-to-Kafka transactions	At-least-once — you implement it
Ordering scope	Best effort, none guaranteed	Per message group	Per ordering key, when enabled	Per queue, single consumer only	Per partition	Whatever your ORDER BY says
Claim / lease	Visibility timeout, extendable per message	Visibility timeout	Ack deadline, extended automatically by client libraries	No timer — unacked messages return when the channel closes	No per-message ack; a stalled consumer is evicted and its partition reassigned	A claimed_until column you manage
Retry control	Redrive policy, maximum receive count	Same	Maximum delivery attempts, retry policy	Manual: nack, or TTL plus a dead-letter exchange	None built in — the application re-produces or diverts	An attempts column and a run_after delay
Dead-letter	Built-in DLQ	Built-in DLQ	Dead-letter topic	Dead-letter exchange via policy	Not built in — you produce to a DLQ topic	A status column
Poison message	Costs one consumer per attempt, then dead-letters	Blocks its message group	Costs one consumer per attempt, then dead-letters	Costs one consumer per attempt	Blocks its whole partition until skipped	Costs one worker per attempt
Transactional enqueue with your write	No	No	No	No	No	Yes — the reason to choose it (The Transactional Outbox)

At-least-once plus idempotent, not exactly-once

The phrase "exactly-once" is doing two jobs and hiding the difference. Delivery is what the broker does with a message. Processing is what your code does to the world. A broker can deduplicate deliveries inside its own boundary; it has no visibility of the charge you made against a payment provider, the email you handed to a mail service or the row you wrote to a database it does not own.

So the correct formulation is: assume at-least-once delivery, and make the consumer idempotent so that repeated delivery produces one effect. That construction is broker-independent, survives retries and rebalances, and does not depend on any product mode you might later turn off (Job Idempotency).

Handling a payment-capture message

Relying on delivery semantics

// "The queue is FIFO with deduplication, so this runs once."
async function handle(msg) {
  await payments.capture(msg.orderId, msg.amountCents)
  await db.orders.update(msg.orderId, { status: 'paid' })
}
// Lease expires mid-capture -> redelivered -> captured twice.
// Broker deduplication never saw the payment provider.

At-least-once delivery, idempotent effect

async function handle(msg) {
  // One row per (message, effect). The unique constraint is the guard.
  const claimed = await db.effects.insertIfAbsent({
    key: `capture:${msg.orderId}`,
    messageId: msg.id,
  })
  if (!claimed) return                       // already done: ack and move on

  await payments.capture(msg.orderId, msg.amountCents, {
    idempotencyKey: `capture:${msg.orderId}`,  // the provider dedupes too
  })
  await db.orders.update(msg.orderId, { status: 'paid' })
}

The second version is correct under every broker, every visibility timeout, every rebalance and every retry, because it does not depend on a delivery guarantee at all. The unique constraint makes the duplicate check atomic — a read-then-write would race with a concurrent duplicate — and the provider-side idempotency key closes the window between claiming the effect and completing it.

The four semantic failures, by symptom

These failures are individually rare per message and statistically certain at volume, which is why they are found by customers rather than by tests. Each maps to a specific mechanism, and recognising the mapping is most of the diagnosis.

The last row is the one that surprises teams: raising consumer concurrency is a routine throughput change, and it silently deletes the ordering guarantee the design depended on.

Semantic failures in production

Trigger	Symptom	Cause	Response
A worker is killed mid-job by a deploy	Two confirmation emails for one order	Work completed, ack never sent, message redelivered	Idempotent consumer plus drain-on-shutdown (Graceful Shutdown)
A dependency is briefly down	Cancelled orders reappear as active	The failed create was retried after the cancel succeeded	Carry a version in the event; ignore anything older than applied state
Jobs slower than usual	Duplicate processing concentrated on the slowest jobs	Processing exceeds the visibility timeout on the tail	Extend the lease inside long handlers; alert on p99 duration versus the timeout
One malformed message	A Kafka partition stops advancing; lag climbs on one key	The consumer throws before committing the offset, forever	Catch, divert to a DLQ topic, commit the offset (Dead-Letter Queues)
Consumer concurrency raised from 1 to 4	Intermittent out-of-order state, only under load	Ordering within a scope requires one consumer for that scope	Partition by entity id so concurrency is across scopes, never within one
A duplicate arriving hours later	A dedupe check that normally works lets one through	Broker deduplication is scoped to a bounded window	Own the deduplication with a retention you chose (Idempotency Storage)

How to build it

Most important first.

Design for at-least-once. Every consumer must be safe to run twice on the same message, before you tune anything else (Job Idempotency).
Do not require ordering unless the business genuinely does. When you do, use the broker's ordering scope — a partition key or message group of the entity id — so that all events for one order are ordered relative to each other and unordered relative to everyone else's.
Prefer designs that are order-insensitive: carry a version or a timestamp in the event and let the consumer ignore anything older than what it has already applied. That survives retries, which ordering scopes do not (Optimistic Concurrency).
Set the visibility timeout above your p99 processing duration, and extend it explicitly inside long-running handlers rather than raising the default for everyone.
Cap attempts and dead-letter. A message that has failed a bounded number of times is a message for a human, not for the fleet (Dead-Letter Queues).
Write down which broker you are on and which guarantee you are relying on, in the code. "The queue is ordered" is a claim that is true of one product's one scope and false everywhere else.

What can go wrong

Failure modes

A duplicate delivery producing a second charge, a second email or a second row, because the consumer assumed once-only.
Events applied out of order, so a cancellation is overwritten by the create that was retried after it.
A visibility timeout shorter than processing, producing concurrent duplicate execution of exactly the slowest jobs.
A poison message blocking a Kafka partition, halting all consumption behind it while depth grows (Queue Backlog).
Ordering silently lost when someone raises consumer concurrency from one to four for throughput.
Deduplication assumed to be global when it is scoped to a window: the same message re-sent after the window is a fresh message to the broker.
A consumer group rebalance mid-batch, so messages already processed but not committed are redelivered to a different member.

What can race

Ack versus lease expiry: the message is redelivered while the first worker is finishing, so two workers hold it simultaneously.
Two workers processing the same entity from different messages, racing on the same row (Backend Races).
A rebalance mid-batch handing already-processed-but-uncommitted messages to another consumer.
A retried older event overtaking a newer one, applying a stale state on top of a fresh one (Optimistic Concurrency).
Deduplication check racing the deduplication write in two concurrent duplicate deliveries — the check must be an atomic insert, not a read-then-write (Duplicate Detection).

Security

A duplicate that charges twice is a financial incident, so idempotency here is a security and compliance control, not only a correctness one (Idempotency in Backends).
Out-of-order application of permission events can leave a revoked user granted: apply a version, not an arrival order, to anything authorization-related (Role-Based Access Control).
A replayed message is an attack if the broker or the network path is reachable. Sign or version messages carrying privileged instructions rather than trusting queue membership as authentication (Webhook Signature Verification).
Dead-letter queues accumulate full payloads of the messages that failed — often the malformed and attacker-shaped ones. Treat DLQ access as production data access (Audit Trails).

Misreads

"Exactly-once delivery." Separate the two claims. A broker may deduplicate deliveries within a bounded scope; your business effect happens once only if your consumer makes it so. Any sentence about exactly-once that does not name that boundary is wrong (At-Least-Once Delivery).
"FIFO means my handlers run in order." FIFO is about the order messages leave the queue. With concurrent consumers, completion order is unrelated (Concurrent Queues).
"Duplicates are a bug in the broker." Duplicates are the guarantee working correctly. The alternative is losing messages.
"We do not need idempotency because our queue is FIFO." Ordering and duplication are independent properties; SQS FIFO still redelivers on visibility timeout expiry.
"Kafka is a queue." It is a partitioned, replayable log. There is no per-message ack, no per-message redelivery and no built-in dead-letter queue — a poison message stops a partition instead of being set aside.

Operating it

How you see it in production

Redelivery rate, as its own metric. A rising rate means leases are expiring, which means duplicates are happening whether or not anyone has noticed.
Processing duration p99 plotted against the configured visibility timeout on the same chart. The crossing point is where duplicate execution starts.
For consumer groups: rebalance frequency. Frequent rebalances mean frequent redelivery (Six Queue Signals, Two That Wake You Up).
Count messages whose delivery attempt is greater than one, by job type. That is your duplicate exposure, expressed as a number.
For ordered scopes: per-key lag rather than aggregate lag. One stalled key is invisible in an aggregate (Depth Is Not an Emergency; Age Is).
Log the message id, delivery attempt and ordering key on every consume, so a duplicate is diagnosable after the fact rather than inferred.

What changes at 10x and 100x

Ordering does not scale. Strict order within a scope means one consumer for that scope, so throughput per scope is capped by one worker regardless of fleet size.
More consumers means more concurrent processing, which means more opportunities for lease expiry, rebalance and redelivery. Duplicate rate rises with fleet size.
At 100x, the ordering key must be fine-grained — per entity, not per tenant — or one busy entity becomes a serial bottleneck for everything sharing its key (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
Deduplication state has to live somewhere, and at high volume that store becomes a system of its own with its own retention and capacity questions (Idempotency Storage).

What this costs

Ordering costs throughput: one consumer per ordered scope, always.
Idempotent consumers cost a deduplication store, a key design and a retention policy — and they buy you freedom from every ordering and duplication concern above.
A long visibility timeout reduces duplicates and increases how long a crashed worker's message is stuck.
Broker-level deduplication reduces duplicate work and gives a false sense of completeness, because it does not cover your external side effects.
Version-based idempotency (ignore anything older than what I have) is more robust than ordering and requires every producer to emit a monotonic version.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

CLOUD-SPECIFICEvery property in this lesson differs by product. SQS standard: at-least-once, best-effort ordering, a per-message visibility timeout you can extend, and a redrive policy to a DLQ after a maximum receive count. SQS FIFO: ordering within a message group, deduplication on a content or explicit id over a bounded window. Google Pub/Sub: at-least-once by default with an ack deadline that client libraries extend for you, optional ordering keys, dead-letter topics after a maximum delivery attempt count, and an exactly-once delivery mode for pull subscriptions. RabbitMQ: consumer acks with no per-message timer at all — unacked messages return only when the channel or connection closes — ordering per queue with a single consumer, and dead-lettering via a dead-letter exchange configured by policy. Kafka: an offset-based log with no per-message ack, ordering per partition, no built-in DLQ, and eviction from the consumer group rather than redelivery when a consumer stalls. Do not carry an assumption from one to another.
SIMPLIFIEDPresents delivery as "one message, one consumer, one attempt at a time". Batching, prefetch and consumer-group rebalancing all complicate this, and each broker complicates it differently.
GENERALOne thing is true everywhere: at-least-once delivery plus an idempotent consumer is the construction that produces once-only business behaviour. Nothing in any product replaces the second half.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designidempotency-vs-deduplication webhook-ordering

Architecturemessage-queues event-driven-architecture

Performancequeue-performance-signals queue-age

Cloudaudit-trails

Concurrencyconcurrent-queues producer-consumer

Domains that do not exist yet

— Distributed Systems — delivery semantics, total versus partial order, and why a general exactly-once guarantee across two independent systems is not available at any price.