Background Jobs
Work that outlives the request that asked for it — and the four guarantees you give up to move it there.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What does moving work out of the request path actually change, beyond making the response faster?
Placing an order should feel instant. Today it takes several seconds because the handler also sends a confirmation email, writes to the search index and calls the warehouse API.
Fire the slow parts off without awaiting them. The handler returns immediately and the work still happens, because the process keeps running.
The process does not keep running on your schedule. A deploy, a scale-in event or an OOM kill ends it, and every un-awaited promise in flight vanishes with no error and no record that it ever existed (Graceful Shutdown).
- The process does not keep running on your schedule. A deploy, a scale-in event or an OOM kill ends it, and every un-awaited promise in flight vanishes with no error and no record that it ever existed (Graceful Shutdown).
- Nothing retries. The warehouse API returns a 503 and the order is simply never dispatched — the customer paid, the handler returned 201, and no system anywhere knows the work is missing.
- Errors have nowhere to go. The exception is thrown after the response was written, so the framework's error middleware never sees it and it surfaces, if at all, as an unhandled rejection warning in a log nobody reads (Error Boundaries: Three Translations, Not One).
- It is unbounded. A burst of orders launches a burst of concurrent outbound calls with no limit, exhausting sockets and the connection pool for the requests that are still in the path (Unbounded Concurrency).
- It is invisible. There is no queue depth, no attempt count, no age — so "did the email go out?" is a question with no answer except searching logs.
What is actually happening
- A background job is work recorded durably, executed later by a process that is not handling a request. The word that carries the weight is *durably*: the record survives the death of the process that created it.
- Fire-and-forget is not a background job. It is a background *task* with no durability, no retry, no visibility and no bound. Those four properties are exactly what a queue adds (Job Queues).
- The move changes the contract. The caller can no longer be told whether the work succeeded, because the response is written before the work runs. The API has to say "accepted", not "done" (The Async Job Pattern).
- It changes the failure surface. A synchronous failure is a status code the caller sees. An asynchronous failure is a message sitting in a retry loop or a dead-letter queue that somebody has to be watching (Dead-Letter Queues).
- It changes consistency. The order row exists before the email is sent, before the index is updated, before the warehouse knows. Every read between those points sees a partially-applied world (Eventual Consistency in Practice).
- And it changes where the work executes. Worker processes have their own deploys, their own resource limits, their own scaling and their own access to secrets — they are a second service that happens to share a repository (Worker Processes).
The same handler, two shapes
The synchronous version is honest about failure and dishonest about time: the caller waits for the warehouse API and learns exactly what happened. The asynchronous version is the reverse. Neither is universally right, and the mistake is not choosing one — it is choosing one without noticing what you gave up.
What makes the second version a background job rather than a detached task is one line: the enqueue happens inside the transaction. That is what makes "the order exists" and "the job exists" a single fact that a crash cannot split.
await db.transaction(async (tx) => {
await tx.insert(order)
})
sendEmail(order) // not awaited
warehouse.dispatch(order) // not awaited
return res.status(201).json(order)
// deploy lands 2ms later: both are gone.
// warehouse 503s: nothing retries, nothing logs.await db.transaction(async (tx) => {
await tx.insert(order)
await tx.insert(outbox, {
type: 'order.placed',
orderId: order.id, // a reference, not a snapshot
correlationId: req.correlationId,
})
})
return res.status(201).json(order)
// the relay publishes committed outbox rows;
// workers retry, and give up into a DLQ someone watches.The second version cannot lose the work to a deploy, cannot act on an order that rolled back, and cannot fail invisibly — because the record of intent is committed atomically with the order and every subsequent step is observable. The cost is that the caller now learns "accepted" rather than "dispatched", which the API contract has to state.
What you actually gain, step by step
It is worth separating the four properties, because teams routinely adopt one and assume the others came with it. A Redis list gives durability and visibility but nothing about bounded concurrency. A thread pool gives bounded concurrency and nothing about durability.
The failure column is the useful one. Each property, absent, produces a distinct and recognisable production symptom.
- 1Durability
The intent is written to storage that survives the process.
fails by Deploy, crash or scale-in silently discards in-flight work.
- 2Atomic with the write
The job record commits with the business row.
fails by A job for an order that rolled back, or an order with no job (The Dual Write Problem).
- 3Retry
A transient failure is attempted again with backoff.
fails by One 503 from a dependency permanently loses the work (Backoff and Jitter).
- 4Bounded concurrency
A fixed worker count caps parallel side effects.
fails by A burst opens unbounded outbound connections and exhausts the pool (Unbounded Concurrency).
- 5Visibility
Depth, age, attempts and outcomes are measurable.
fails by "Did the email send?" has no answer except grepping logs.
- 6Terminal handling
Work that cannot succeed lands somewhere a human sees.
fails by Infinite retry of a poison job, consuming capacity forever (Dead-Letter Queues).
A queue product gives you three to five of these. Atomicity with your database write is the one it can never give you — that one is yours (The Transactional Outbox).
How background work fails in production
Async failures do not look like errors. They look like absence: an email that never arrived, a search result that is missing, a report that was requested and forgotten. The reports arrive days later and from customers rather than from alerts.
Every row here is preventable by something in the design list, which is why the design list is worth treating as a checklist rather than as advice.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A deploy | A batch of confirmation emails never sent | Detached tasks in memory, killed with the process | Durable enqueue plus drain-on-shutdown for in-flight jobs (Graceful Shutdown) |
| A dependency returns 503 briefly | A handful of orders never reach the warehouse | No retry — one failure is terminal | Retry with backoff and jitter; dead-letter after a bounded number of attempts |
| A worker starts immediately after enqueue | Intermittent "order not found" in worker logs | Enqueued before the transaction committed | Outbox, or enqueue in an after-commit hook |
| A traffic spike | API latency rises although the work is async | The enqueue itself is a synchronous call to a slow broker | Treat the broker as an external dependency: timeout, and decide the fallback (Timeouts) |
| One bad payload | Worker CPU pinned, unrelated jobs delayed for hours | Poison job retried without limit | A max attempt count and a dead-letter queue |
| Nothing in particular | A user says a report from Tuesday never arrived | Job failed, was swallowed, acked anyway | Ack only after success; count outcomes by type; alert on the DLQ |
How to build it
Most important first.
- Record the intent durably inside the transaction that produced it, then execute outside. That is the outbox pattern, and it is the only way to make "the order exists" and "the job exists" one atomic fact (The Transactional Outbox).
- Enqueue a reference, not a payload snapshot: the order id, not a serialized copy of the order. The worker reads current state when it runs, so a job that sits for ten minutes does not act on ten-minute-old data.
- Make every job idempotent before you make it retryable. Retries are the point of a queue, and a queue that retries a non-idempotent job sends two emails (Job Idempotency).
- Give the caller a way to observe the outcome — a job id and a status endpoint, a webhook, or a record on the resource itself. "It happened somewhere" is not a product behaviour (Long-Running Operations: 202 and the Job Resource).
- Run workers as their own deployable process. Sharing a process with the HTTP server means a CPU-heavy job starves request handling, and it means you cannot scale the two independently (Worker Scaling).
- Instrument from day one: queue depth, oldest message age, attempts, duration, outcome. Async work that is not measured is async work that has already failed silently at least once (Six Queue Signals, Two That Wake You Up).
What can go wrong
- Jobs lost on deploy because the process was killed with work in memory and nothing durable behind it.
- Jobs that succeed at the database and fail at the side effect, leaving a committed row and an email that was never sent — or the reverse, if the job runs before the commit (The Dual Write Problem).
- A payload that references a row that has since been deleted, so every retry fails identically until it dead-letters.
- Jobs enqueued by a transaction that later rolled back, so the worker acts on an order that does not exist.
- Poison jobs retried forever, consuming worker capacity that legitimate work needed (Queue Backlog).
- A worker deploy that changes the payload schema while old-format jobs are still in the queue (Expand and Contract Migrations).
- Silent success: the job ran, threw, was swallowed by a bare
catch, and was acked anyway.
- Enqueue-before-commit: the worker starts, reads the row, and it is not there yet. Common, intermittent, and almost always misdiagnosed as a broker problem.
- Two workers claiming the same job because a lease expired while the first was still working (Queue Semantics).
- A job and a user action racing on the same row — the export job reads a record the user deletes mid-run (Backend Races).
- A worker deployed with a new payload schema while the previous version is still consuming, so two decoders race over one queue.
- A job payload is untrusted input to the worker, exactly like a request body. It may have been enqueued by an older version, tampered with if the broker is reachable, or crafted if any user input reaches it unvalidated (The Three Validations).
- Re-establish authorization inside the worker from the stored actor id. The worker runs with no HTTP session, and "the request that enqueued this was authorized" does not mean the action is still permitted when it executes (Where the Check Belongs).
- Never put secrets, tokens or full personal records in a payload. Queue contents are frequently logged, dumped during triage, and retained in a dead-letter queue for days (Secrets in Logs).
- Workers usually hold broader credentials than the API — they write to more systems and call more third parties. Scope those credentials per job type rather than giving every worker the union of all of them (Least Privilege in Infrastructure).
- A URL in a payload that the worker fetches is a server-side request forgery primitive that bypasses every check the API layer performed (SSRF — When the Backend Fetches a URL).
- "Async is more scalable." Async moves work in time; it does not reduce it. If the work arrives faster than workers complete it, you have a growing backlog instead of a slow API — and a backlog is harder to see and harder to recover from (Queue Backlog).
- "Fire-and-forget is a background job." It has none of durability, retry, visibility or bounded concurrency. It is the request path with the error handling removed.
- "The job will retry, so failures are handled." Retries help transient failures. A bug, a bad payload or a deleted row fails identically every time, forever, until something dead-letters it.
- "Once it is in the queue it is safe." Enqueue-outside-the-transaction can lose the job when the commit fails, or run it against a row that never committed (The Transactional Outbox).
- "Workers are just the same app in a different mode." They are a second service: different scaling, different failure modes, different credentials and a different deploy that can be a version behind.
Operating it
- Queue depth and oldest message age. Depth alone is ambiguous — a thousand messages that arrived one second ago is healthy; ten that arrived an hour ago is an incident (Depth Is Not an Emergency; Age Is).
- Attempts per job. A rising attempt distribution is a dependency degrading, visible before any job dead-letters.
- Job duration by type, and outcome counts by type. Aggregating across types averages a 50ms email with a 20-minute export and describes neither.
- End-to-end latency: enqueue timestamp to completion timestamp. That is what a user experiences, and it is invisible in worker-side duration metrics.
- Propagate the correlation id from the enqueuing request into the payload, so one identifier joins the request log and the worker log (Correlation Ids That Survive Every Hop).
- Alert on the oldest-message age crossing the point where the delay becomes user-visible, not on depth.
- At 10x, background work is what lets the request path stay flat: the API stays fast and the queue absorbs the variance. This is the main reason to do it.
- At 10x, the worker fleet becomes the constraint instead of the API. Sizing it is a separate capacity question with a separate signal (Worker Scaling).
- A shared queue means one slow job type delays every other. Above a certain volume, separate queues per job class stop being tidiness and start being isolation (Bulkheads).
- At 100x, the enqueue path itself matters: a broker write on every request is a dependency on the request path, and if it is slow, your fast API is now as slow as your queue.
- You trade an immediate, precise answer for a fast response. The caller learns that the work was accepted, not that it worked.
- You trade one failure domain for two, and the second one has no user watching it — which is why dead-letter monitoring is mandatory rather than nice to have.
- You accept eventual consistency in every read that happens between the commit and the job. Some of those reads are the user's own next page load.
- You take on a broker, workers, deploys, retries and dashboards. For work that finishes quickly and rarely fails, that is a large amount of machinery for a small saving (Request or Background?).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALDurability, retry, visibility and bounded concurrency are what distinguish a job from a detached task, in any language or stack.
- RUNTIME-SPECIFICWhat "fire and forget" does differs. In Node an un-awaited promise that rejects becomes an unhandled rejection which, depending on version and flags, can terminate the process. In Python a detached asyncio task that is not referenced can be garbage-collected mid-flight and its exception never surfaces. In a pre-fork worker model the task dies with whichever worker happened to serve the request.
- CLOUD-SPECIFICOn serverless platforms there is often no "after the response" at all — the execution environment can be frozen the instant the handler returns, so detached work simply does not run. Background work there must go to a queue or a separate invocation by construction (Serverless Backends).
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — why "the row committed and the message was sent" is not one event, and what the alternatives to a shared transaction actually guarantee.