Background Jobs and Workers

The request path is the most expensive place to do anything: the user is waiting, a connection is held, the load balancer has a timeout ticking, and any failure surfaces as an error the user sees. Work that the user does not need to *see* finish — sending the confirmation email, resizing the uploaded image, rendering the monthly report, calling a model that takes 20 s, transcoding a video — belongs elsewhere. The API writes a job to a queue and returns 202 Accepted with a job id; a fleet of workers consumes the queue and does the work at its own pace.

This is the same decision as Request/Response vs Event-Driven, applied to work rather than to calls. The gain is decoupling in three dimensions: latency (the API responds in milliseconds), failure (a transcoder outage becomes a growing backlog, not a stream of 500s), and capacity (workers scale on queue depth, the API scales on request rate). The cost is that the job now has a lifecycle you must manage, and that the user has to be told what happened later.

A job is a row (or a message) with a state. It is queued when the API writes it, running when a worker takes it, succeeded or failed when the worker finishes, and — the state people forget — retrying between a failure and the next attempt. The worker takes a job with a lease (a visibility timeout in SQS terms, a locked_until column in a database queue): if the worker dies mid-job the lease expires and another worker picks the job up. That is what makes the system survive worker crashes, and it is also why every job runs at least once, never exactly once — a worker that finished the work but died before acknowledging leaves a job that will run again.

So every handler must be safe to run twice. Sending an email twice is annoying; charging a card twice is an incident. Handlers achieve this by keying side effects on the job id (the email provider gets idempotency_key = job.id), by making the write an upsert, or by checking a processed_jobs table inside the same transaction as the work — the patterns are the subject of the next lesson. Timeouts bound a single attempt: a transcoding job that normally takes 6 s should be killed at 60 s, marked failed, and retried, rather than holding a worker forever.

Workers pull with a bounded concurrency — 4 transcodes per 8-core box, 50 emails per worker because the provider allows 100 req/s across the fleet. Concurrency limits are the worker-side half of Backpressure: without them a burst of 10,000 jobs turns into 10,000 simultaneous calls to a provider that will rate-limit or fall over. Limits are often per *resource* rather than per worker: a global semaphore in Redis (INCR with a cap, see Redis: Data Structures, Not a Cache) caps calls to one external API across all workers.

Priorities exist because a queue is FIFO and not all jobs are equal: a password-reset email must not wait behind 50,000 newsletter sends. The cheap solution is separate queues per priority class, each with its own workers, so a flood in one cannot starve another; a single Priority Queue is the in-memory version of the same idea. Scheduled jobs — run at 03:00, run every 5 min — are a scheduler that *enqueues* at the right time, not a worker that sleeps; that keeps the execution path identical and makes a missed schedule visible as an empty queue rather than a silent no-op. Jobs with dependencies (thumbnail after transcode, index after upload) form a small DAG (Directed Acyclic Graph) and run in Topological Sort order; most systems model this as a job that enqueues its successors on success.

The user submitted a video and got a job id. Now what? Two options. Polling: the client calls GET /jobs/{id} every few seconds and reads the state — simple, works through any proxy, and costs one cheap read per poll; back off the interval as the job ages. Push: the server notifies the client over a WebSocket, Server-Sent Events, or a webhook when the state changes — better latency and no polling load, at the cost of a stateful connection or a webhook delivery problem of its own (see API Architecture: REST, GraphQL, RPC, gRPC, WebSockets, Webhooks). Most products poll first and add push when polling load or perceived latency becomes a measured problem. Either way, the job row is the source of truth; the notification is a hint to go read it.

Workers scale on queue depth and oldest-message age, not on CPU. If the queue holds 20,000 jobs and the oldest is 15 min old, add workers; if the queue is empty, remove them. Autoscaling on that signal is the point of the whole design — the API fleet never needs to know how many workers exist. The limit is downstream: adding workers does nothing if they all wait on the same rate-limited provider or the same database, and that is where the queue starts hiding a capacity problem rather than absorbing a burst.