Job Queues
Enqueue, claim, process, ack, retry, dead-letter — the six-step lifecycle every queue implements, however it spells them.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What happens to a message between being enqueued and being finished with, and which step is the one that loses work?
Deferred work must survive a restart, be picked up exactly by one worker at a time, retry when a dependency is down, and stop retrying when it never will succeed.
Push job ids onto a Redis list and have workers LPOP them in a loop. It is a queue, it is durable enough, and it took ten minutes to write.
A worker pops a job and then crashes. The job is gone: it was removed from the list at pickup, and nothing recorded that it was ever claimed. This is the single most common way home-grown queues lose work.
- A worker pops a job and then crashes. The job is gone: it was removed from the list at pickup, and nothing recorded that it was ever claimed. This is the single most common way home-grown queues lose work.
- There is no attempt count, so a job that fails is either retried forever by whatever re-enqueues it, or lost. Neither is a decision anyone made.
- There is no visibility timeout, so a worker that hangs holds the job invisibly and no other worker can take it — the job is neither in flight nor available.
- Nothing separates "failed once" from "will never succeed". A malformed payload retried in a tight loop consumes the whole fleet (Dead-Letter Queues).
- The enqueue is a second write outside your transaction, so an order can commit with no job, or a job can exist for an order that rolled back (The Dual Write Problem).
What is actually happening
- Every queue — managed, self-hosted or built on your database — implements the same six steps. What differs is the vocabulary and which steps you have to write yourself.
- Enqueue writes the message durably and returns. Claim hands it to exactly one consumer and starts a lease. Process runs your code. Ack tells the broker the work is done and it may delete the message. Retry returns an un-acked message to the queue after a delay. Dead-letter moves a message that has exhausted its attempts somewhere terminal.
- The claim step is where the guarantees live. A broker does not remove a message when a worker takes it; it makes it *invisible* to other consumers for a lease period. If the ack does not arrive before the lease expires, the message becomes visible again and someone else gets it.
- That is precisely why duplicates are normal. A worker that finishes the work and dies before acking has done the job once and will have it delivered again (At-Least-Once Delivery).
- A database-backed queue implements the same six steps with a table: a
statuscolumn, aclaimed_untiltimestamp as the lease, anattemptscounter, andSELECT ... FOR UPDATE SKIP LOCKEDto claim without two workers colliding. It is slower than a purpose-built broker and it has one property none of them has — the enqueue can be in the same transaction as your business write (The Transactional Outbox). - Push versus pull changes who controls the rate. Pull (a worker asking for the next message) gives you natural backpressure: workers take what they can handle. Push (the broker calling your endpoint) requires the broker to respect a concurrency limit you configure, and gives you a throughput problem when it does not (Backpressure).
The six steps, and what each one loses
Learn the lifecycle rather than a library. Every broker implements these six steps; when you switch products, what changes is which of them are configured, which are automatic, and what the defaults are.
Two steps do all the damage. Ack decides whether a crash loses work or duplicates it. The lease decides whether a stuck worker blocks a message or hands it to a second worker while the first is still running.
- 11. Enqueue
Writes the message durably; returns to the producer.
fails by Outside the producing transaction, so the job and the row can disagree (The Dual Write Problem).
- 22. Claim
Hands the message to one consumer and starts a lease; the message becomes invisible to others.
fails by Lease too short: a second worker gets it while the first still runs. Too long: a crashed worker blocks it.
- 33. Process
Runs your handler.
fails by Slow dependency, bad payload, deleted referent, or a bug that fails identically every attempt.
- 44. Ack
Tells the broker to delete the message.
fails by Acking at claim time loses work on crash; acking after the effect but before a crash duplicates it.
- 55. Retry
Returns an un-acked or explicitly failed message after a delay.
fails by No backoff hammers a struggling dependency; no cap retries a poison message forever (Backoff and Jitter).
- 66. Dead-letter
Moves a message past its attempt limit somewhere terminal and visible.
fails by No DLQ at all, or a DLQ nobody has ever looked at (Dead-Letter Queues).
Steps 2 and 4 together are why at-least-once is the normal guarantee: the broker cannot tell a worker that died before acking from a worker that died before doing the work.
Where duplicates come from, drawn
The message is not removed at claim time — it is hidden. Everything that follows from that is visible in one picture: the lease is a timer, and if the ack loses the race with it, the message comes back.
Notice that the broker behaves identically whether the worker crashed before doing the work or after. It has no way to distinguish them, which is why the correctness burden lands on the worker (Job Idempotency).
A queue in your database, if you already have one
A table with a status, a lease and an attempt count is a real queue with one property no broker can offer: the enqueue is part of your transaction, so the job and the row it refers to commit or roll back together. For a service that already runs PostgreSQL and has moderate volume, this removes the dual-write problem instead of working around it.
SKIP LOCKED is the load-bearing clause. Without it, concurrent claimers block on each other's locked rows and the queue serialises; with it, each worker takes the next row nobody else has locked and contention stays low. The claimed_until column is the lease, and the recovery query for crashed workers is the same query — a row whose lease has passed is available again.
1-- Claim up to 10 due jobs, extend a 5 minute lease, and return them.2-- SKIP LOCKED lets N workers run this concurrently without blocking.3UPDATE jobs4SET status = 'claimed',5 claimed_until = now() + interval '5 minutes',6 attempts = attempts + 17WHERE id IN (8 SELECT id FROM jobs9 WHERE (status = 'pending' AND run_after <= now())10 OR (status = 'claimed' AND claimed_until < now()) -- crashed worker11 ORDER BY run_after12 FOR UPDATE SKIP LOCKED13 LIMIT 1014)15RETURNING id, type, payload, attempts;The second WHERE branch is the recovery path: a claimed row whose lease has expired is indistinguishable from a pending one, so a crashed worker needs no separate reaper. The attempt increment happens at claim, not at failure, so a worker that crashes still counts its attempt and cannot loop forever.
How to build it
Most important first.
- Ack after the work succeeds, never at pickup. Acking on receipt converts every worker crash into lost work, and it is the default in more client libraries than you would expect.
- Set the lease longer than your worst realistic job duration, and extend it explicitly for long jobs rather than setting a very long default — a long default means a crashed worker's job is stuck for that long (Queue Semantics).
- Store the attempt count on the message and cap it. Unbounded retry is not resilience; it is a way to spend your entire fleet on one bad payload.
- Use exponential backoff with jitter between attempts, so a dependency coming back up does not immediately receive every failed job at once (Backoff and Jitter).
- Separate queues by job class and latency expectation. A single queue means a bulk export blocks a password-reset email behind it (Bulkheads).
- Start with a database-backed queue if you already have a transactional database and modest volume. It removes the dual-write problem entirely and it is one fewer system to operate; move to a broker when volume, fan-out or cross-service consumption justifies it.
- Make the payload a reference plus a version tag, so a worker running old code can recognise a message it does not understand instead of misinterpreting it.
What can go wrong
- Ack-on-receipt: the message is deleted at claim time, so a crash mid-processing loses the work permanently.
- A lease shorter than the job duration: the message becomes visible again while the first worker is still running it, and two workers process it concurrently (Job Idempotency).
- A lease much longer than the job duration: a crashed worker's message is invisible for the whole lease before anyone can retry it.
- Unbounded retry of a poison message, saturating the fleet (Queue Backlog).
- Retry with no backoff, producing a tight failure loop against a dependency that is already struggling (Retry Storms).
- A worker killed during a deploy without draining, dropping every in-flight message back to the queue at once (Graceful Shutdown).
- Payload schema changed while old messages are still queued, so the new worker throws on every one of them (Expand and Contract Migrations).
- Two workers claiming the same message when a lease expires mid-processing — the defining race of every queue.
- Claim contention in a database-backed queue when two workers select the same row;
FOR UPDATE SKIP LOCKEDexists specifically to make this deterministic (Pessimistic Locking). - Ack racing lease expiry: the ack arrives just after the message was redelivered, so one worker acks a message another worker now owns.
- Enqueue racing commit: the worker claims and processes before the producing transaction is visible (The Transactional Outbox).
- A shutdown racing an in-flight job: the process exits between the work completing and the ack being sent, guaranteeing a duplicate (Graceful Shutdown).
- Restrict who may enqueue. A queue reachable by anything that can reach the network is an unauthenticated remote code path into your workers (Public Exposure, Read With Context).
- Validate the payload in the worker as untrusted input. It may come from an old version, a different service, or a message someone wrote directly to the broker (Transport Validation).
- Carry the actor id in the payload and re-check authorization at execution time; do not carry a token or a session, which will be expired or over-privileged by the time the job runs (Object-Level Authorization).
- Dead-letter queues retain payloads for days by design. Anything sensitive in a message is sensitive data sitting in a place with looser access controls than your database (Secrets in Logs).
- "The queue guarantees the job runs once." It guarantees the message is delivered at least once in the normal case. Running the *effect* once is your job (Job Idempotency).
- "Acking early is fine, the work is nearly done." The window between ack and completion is exactly the window in which work is lost, and it is the window a deploy lands in.
- "Redis is a queue." A list is a data structure; a queue is a lifecycle. Redis-based job libraries add the claim, lease, attempt and dead-letter machinery on top — a bare
LPOPloop has none of it (Circular Queue). - "More workers means faster drain." Only until the shared dependency saturates. Past that, more workers make everything slower including the requests still in the path (Connection Pools).
- "A retry means the job failed." A retry can mean the job succeeded and the ack was lost. Those are indistinguishable from the broker's side, which is the whole reason idempotency is required.
Operating it
- Depth, oldest-message age, in-flight count and dead-letter count — four numbers, and only the second one tells you whether users are affected (Six Queue Signals, Two That Wake You Up).
- Attempt-count distribution. A shift from mostly-one to mostly-three is a dependency degrading, long before anything dead-letters.
- Processing duration by job type against the configured lease. Jobs approaching the lease are duplicate deliveries waiting to happen.
- Ack rate versus claim rate. A persistent gap means work is being claimed and not completed.
- For a database-backed queue: rows in
claimedstatus with an expired lease. That count is your crashed-worker rate, and no other signal shows it. - Log claim, success and failure with the message id, the attempt number and the correlation id from the enqueuing request (Correlation Ids That Survive Every Hop).
- At 10x, a database-backed queue starts to show: claim contention, index bloat on the status column, and vacuum pressure from high row churn. That is the signal to move to a broker, not the volume itself.
- At 10x, per-message overhead starts to dominate for small jobs. Batching — claiming and acking many messages at once — is the usual answer, and it makes partial failure within a batch a new problem to solve.
- At 100x, one queue becomes many, partitioned by job class and often by tenant, so that a large customer's bulk work cannot starve everyone else (Multi-Tenancy).
- Worker count is bounded by what the *downstream* dependency can take. Scaling consumers past the database's capacity converts a queue backlog into a database outage (Worker Scaling).
- A database-backed queue removes the dual-write problem and adds load to the database you were trying to protect.
- A managed broker gives you durability, leases, retries and dead-lettering for free, and puts your enqueue outside your transaction — so you need an outbox anyway if losing a job is unacceptable.
- Long leases reduce duplicate processing and increase how long a crashed worker's message is stuck.
- Many small queues give isolation and multiply the number of things to monitor, scale and alert on.
- Batching improves throughput and makes the failure story worse: one bad message in a batch of a hundred needs a partial-failure path.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe six-step lifecycle describes every queue; only the vocabulary and the defaults change.
- CLOUD-SPECIFICThe claim step has different names and different mechanics: SQS calls the lease a visibility timeout and lets you extend it per message; Pub/Sub calls it an ack deadline and its client libraries extend it automatically while your handler runs; RabbitMQ has no timer at all and redelivers unacked messages only when the channel or connection closes; Kafka has no per-message ack — a consumer that stops calling poll is evicted from the group and its whole partition is reassigned. Code written against one of these does not transfer.
- DATABASE-SPECIFIC
SELECT ... FOR UPDATE SKIP LOCKEDis available in PostgreSQL 9.5+ and MySQL 8.0+. Older versions and other engines need a different claim strategy — an UPDATE-then-SELECT on a claim token, or advisory locks — with different contention behaviour.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — delivery semantics and the consensus problem underneath a broker that claims stronger guarantees than at-least-once.