Engineer Atlas
OverviewLearnArchitecture FinderFailure SimulatorDiagram BuilderRoadmapPracticeInterview
OverviewLearnArchitecture FinderFailure SimulatorDiagram BuilderRoadmapPracticeInterviewCheat SheetCompareConnections
Software Architecture
  • Architecture Fundamentals
  • Application Architecture
  • Code Architecture
  • Events & Messaging
  • APIs & Gateways
  • Scaling & Caching
  • Async Processing
  • Distributed Data
  • Reliability
  • Distributed Systems
  • Observability
Architecture/Learn/Async Processing

Async Processing

Background jobs, workers, idempotency under retries, and backpressure when producers outrun consumers.

Background Jobs and Workers
▶ interactive

Move work that does not have to finish inside the request — image processing, email, reports, AI generation, transcoding — onto a queue consumed by workers, and accept that a job which can be retried will eventually run twice.

Idempotency
▶ interactive

A network gives you at-least-once delivery whether you like it or not; idempotency — the same request applied twice has the effect of once — is what turns "at least once" into the behaviour users mean by "exactly once".

Backpressure
▶ interactive

When a producer emits 100,000 messages per second and the consumer handles 20,000, the queue grows by 80,000 per second and the oldest message is minutes old within minutes; backpressure is every mechanism that makes the producer feel the consumer's limit before memory, disk or latency does.

Engineer Atlas
GitHub·LinkedIn