EventsGENERALSCALE-SPECIFICSIMPLIFIED

Event-Driven Backends

Publishing a fact instead of calling the next step, what that actually buys, and the honest list of what it costs.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What does my backend gain, and pay, by publishing an event instead of calling the next step directly?

The requirement

Checkout has grown to seven downstream steps. It is slow, it fails whenever any of the seven is unwell, and every new feature request means another edit to the same handler.

The obvious build

Keep calling each dependency inline from the order handler. It is explicit, it is easy to read, and everything is in one place.

Why it breaks

Checkout latency is the sum of all seven calls, and its p99 is dominated by whichever dependency is worst at that moment (Why Is My API Slow?).

How it breaks in production

Checkout latency is the sum of all seven calls, and its p99 is dominated by whichever dependency is worst at that moment (Why Is My API Slow?).
Checkout availability is the product of all seven. Seven dependencies at 99.9% each is well under 99.9% for the thing customers actually use.
The analytics service has a bad deploy and orders stop being accepted, because a non-essential call was made a required one by being in the same code path.
Each inline call holds a request-handling worker and, if the transaction is still open, a database connection while it waits on the network (External Calls Inside a Transaction, Connection Pool Exhaustion).
Adding the eighth step means editing, reviewing and redeploying the most business-critical handler in the system.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

The write commits a fact. A separate step publishes that fact to a broker. Consumers subscribe and do their own work on their own schedule, with their own retries and their own failure budget.
What actually changed is coupling in time and in availability. The producer no longer waits for, or depends on, the consumer being up. It still depends on the broker being up — the dependency was replaced, not removed.
Bursts are absorbed rather than rejected. A queue lets a spike of 10,000 orders drain against a consumer that processes 500 a second, instead of 10,000 requests each holding a worker (Backpressure).
The commit and the publish are two different systems, so they can disagree. Committing then publishing can lose the event on a crash; publishing then committing can announce a fact that then rolls back (The Dual Write Problem).
Failure changes shape rather than disappearing. A synchronous failure is loud, attributable and immediate. An asynchronous failure is a growing backlog, a dead-letter queue and a customer asking why they never got their receipt (Queue Backlog).

The publish path, step by step

The interesting engineering is not in the consumer. It is in the four steps between "the write committed" and "a consumer has the event", because every one of them can fail in a way that produces no error anywhere.

Read the pipeline below as a list of things you must have an answer for. The design that has an answer for every row is the outbox; the design that has an answer for none of them is await broker.publish() immediately after await tx.commit().

From committed fact to consumed event

1
Write commits
The fact becomes true in the database.
fails by Rolls back after the event was already published — consumers act on nothing.
2
Outbox row
The event is written in the same transaction as the fact.
fails by Skipped entirely, which is the dual-write bug (The Dual Write Problem).
3
Relay publishes
A separate process reads unpublished outbox rows and sends them to the broker.
fails by Relay stopped and nobody noticed; publishes duplicated after a crash between send and mark-sent.
4
Broker accepts and retains
Durably stores the event for its subscribers.
fails by Retention shorter than a consumer outage, so recovery loses events permanently.
5
Consumer receives
Pulls or is pushed the event, does its work.
fails by Crashes after the side effect and before the ack, so the event is redelivered (At-Least-Once Delivery).
6
Consumer acks
Tells the broker this message is done.
fails by Acked before the work completed, losing the event on a crash — the opposite and worse mistake.

Note the asymmetry: acking too late causes duplicates, which idempotency solves. Acking too early causes loss, which nothing solves.

What moved to the consumer

A synchronous call fails in one place and you know immediately. An event-driven flow relocates each of those failures into a different process with different alerting, and the table below is the translation you need in your head before you commit to the pattern.

The row that surprises teams is the last one. In a synchronous system, "did it happen?" is answered by a status code. In an event-driven one, the honest answer requires reconciliation — comparing what should have happened against what did (Keeping a Search Index in Sync).

The same failure, before and after

Trigger	Symptom	Cause	Response
Email provider returns 500	Synchronous: checkout returns 500 and the order is lost. Event-driven: order succeeds, email retried later.	A non-critical dependency was in the critical path.	Move it to a consumer; the response should never depend on it.
Consumer deployed with a bug	No upstream error at all. Backlog grows; one product capability silently stops.	The producer has no feedback path from consumers, by design.	Alert on oldest-message age per consumer group, not on error rate alone (Queue Backlog).
Broker unavailable	Publishes fail. If publishing is inline in the request, every write now fails.	The broker became a hard synchronous dependency of the write path.	Outbox: the write commits regardless, the relay catches up when the broker returns (The Transactional Outbox).
Poison message	One consumer retries forever, blocks its partition, and processes nothing else.	Retry without a bound or a dead-letter path.	Bounded retries, then dead-letter with the payload and error preserved (Dead-Letter Queues).
Consumer slower than producer for an hour	Everything works; results are just old. Users see stale search results and late emails.	Consumer throughput below sustained publish rate.	Scale consumers on lag, not on CPU. Lag is the signal that matches user experience (Worker Scaling).
Someone asks "did every order get an invoice?"	Nobody can answer without a query neither system was designed to support.	No reconciliation between the fact and the reaction.	A periodic job that compares source-of-truth rows to consumer outcomes and reports the gap.

Publishing without lying

DATABASE-SPECIFICThe outbox needs the event row and the business row in one transaction, which assumes a database with multi-row atomic commits. On a store without cross-document transactions the equivalent is change-data-capture from the store's own log, which gives similar guarantees with different operational ownership.

Almost every serious event-driven bug reduces to the same thing: the event and the state disagreed. The version below is the shape that does not lie, and it is deliberately unexciting — one transaction, one table, one relay.

The cost is visible in the code: an extra table, a background process, and duplicate publishes on relay crashes that consumers must absorb. That last one is not a flaw to be fixed; at-least-once is what you get, and idempotent consumers are the price of admission (Job Idempotency).

Announcing an order

Dual write

await db.transaction(async (tx) => {
  await tx.orders.insert(order)
})
// crash here: order exists, nobody ever hears about it
await broker.publish('OrderPlaced', { orderId: order.id })
// or publish first and roll back: consumers act on nothing

Outbox, then relay

await db.transaction(async (tx) => {
  await tx.orders.insert(order)
  await tx.outbox.insert({
    id: eventId,              // stable: consumers dedupe on it
    type: 'OrderPlaced',
    payload: { orderId: order.id, version: 1 },
    correlationId: ctx.correlationId,
  })
})
// separate relay process:
//   read unpublished rows in order, publish, mark sent.
//   a crash between publish and mark re-sends -> duplicates,
//   never loss.

The fact and its announcement now share one atomic commit, so they cannot disagree. The remaining failure mode is duplicate delivery, which a consumer can handle deterministically with the event id — whereas a lost event has no local fix at all.

How to build it

Most important first.

Split the flow by what the response depends on. Charging the card belongs in the request; the receipt email, the search index update and the analytics row do not (Request or Background?).
Publish through a transactional outbox so the fact and its announcement commit atomically, and a relay does the actual publishing (The Transactional Outbox).
Make every consumer idempotent before you make anything else fast. At-least-once delivery is the normal case, not the edge case (Job Idempotency, At-Least-Once Delivery).
Give each consumer its own retry policy and dead-letter queue, so a poison message in analytics cannot stall the email consumer (Dead-Letter Queues).
Monitor consumer lag as a first-class product signal. In a synchronous system "it is broken" is an error rate; here it is an age (Queue Backlog).
Keep the synchronous path when it is genuinely simpler. A single-deployable system with an in-process dispatcher gets most of the decoupling and none of the delivery problems (The Modular Monolith).

What can go wrong

Failure modes

Event published, transaction rolled back: consumers act on an order that does not exist. This is the single most common event-driven bug and the outbox exists for it.
Transaction committed, publish failed: the order exists and nothing downstream ever happens. Silent, and only visible by reconciling.
A consumer that has been dead for six hours. Nothing errors upstream; the backlog simply grows and the product quietly stops doing one of its jobs.
The broker itself failing, which now takes out every reaction in the system at once — a shared dependency with the blast radius of a database (Cascading Failure).
Retries in the consumer hammering a dependency that is already down, turning a partial outage into a full one (Retry Storms).
Ordering assumed and not guaranteed: OrderCancelled processed before OrderPlaced, leaving an order that is live and should not be (Writing Event Consumers).

What can race

The event arrives at a consumer before the producing transaction is visible on the read replica the consumer queries — a fact that is true and not yet readable (Eventual Consistency in Practice).
Two events for the same entity processed concurrently by two consumer instances, applying updates in the wrong order (Optimistic Concurrency).
A retry of an event overlapping with the original attempt that has not actually failed, producing two concurrent executions of the same reaction (Duplicate Detection).

Security

The broker is now a trust boundary. A consumer must validate event payloads exactly as it would validate an HTTP body — an event from a compromised or buggy producer is untrusted input (The Trust Boundary).
Events crossing tenant lines is a real risk: a consumer that processes all events must apply the tenant scope itself, because the topic has no per-tenant boundary (Tenant Isolation).
Broker credentials are production credentials with fan-out reach. Topic-level publish and subscribe rights should be least-privilege per service (Secrets Are Not Configuration).

Misreads

"Async is more scalable." Not by itself. Async decouples availability and smooths bursts; it does not reduce the work, and a consumer pool sized wrong is exactly as overloaded as a synchronous service was — it just fails as latency instead of as errors.
"The response is fast now, so the system is faster." The user-visible completion time may be unchanged or worse. You moved the wait from a spinner to an inbox.
"Event-driven means microservices." An event dispatcher inside one process is event-driven and needs no network at all (The Modular Monolith).
"We use a managed broker, so delivery is exactly-once." Brokers can offer strong delivery guarantees within their own boundary; your handler still has to be idempotent, because the effect of processing lives in your database, not in the broker (Job Idempotency).
"Events are fire and forget." Fire and forget is a decision to not know whether it happened. Almost nothing in a business system can afford that, which is why reconciliation exists.

Operating it

How you see it in production

Per topic: publish rate, per-consumer processing rate, and age of the oldest unprocessed message. Age is the signal that maps to customer experience; depth does not.
Dead-letter queue depth with an alert at any non-zero sustained value, plus the first failing message's payload and error.
End-to-end trace linking the producing request to each consumer's work through the correlation id carried in the event envelope (Tracing From the Backend's Side).
A reconciliation count: rows created in the last hour versus events consumed for them. A persistent gap is a lost-publish bug (Keeping a Search Index in Sync).

What changes at 10x and 100x

Consumers scale independently. The email consumer can run two workers while the indexer runs twenty, which is not possible when both are lines in one handler (Worker Scaling).
The request path stops growing with feature count, which is the durable benefit — checkout latency is unchanged by the eleventh consumer.
Total work does not fall. It moves. You are now provisioning consumer capacity, broker capacity and retry capacity, and the bill and the operational surface both grow.
At very high fan-out, the payload size multiplies by consumer count on the network and in every consumer's storage, which makes fat events expensive in a way thin ones are not (Naming Events).

What this costs

You trade a stack trace for a distributed trace. Debugging "the customer did not get their email" goes from reading one function to inspecting a broker, a consumer group, a dead-letter queue and three logs.
You trade immediate failure for delayed, silent failure. Synchronous errors are annoying and obvious; backlogs are comfortable and invisible until they are not.
You trade one deployable for several with a schema contract between them. Changing an event payload now requires a compatibility plan.
Local development and testing get harder: reproducing a flow means running a broker and consumers, or building a test harness that fakes them convincingly (Test Against the Real Database).

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe coupling and failure-shape arguments hold for any broker and any language.
SCALE-SPECIFICBelow roughly a handful of downstream reactions in a single deployable, an inline call or an in-process dispatcher is simpler, fully traceable and has no delivery problem. The costs listed here are paid from the first event; the benefits arrive with the fourth or fifth consumer.
SIMPLIFIEDTreats "the broker" as one thing. Real brokers differ substantially in ordering scope, retention, replay and consumer-group semantics, and those differences change which designs here are even available (Queue Semantics).

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designasync-job-pattern long-running-operations partial-failure

Architectureevent-driven-architecture message-queues sync-vs-async

Performancequeue-age queue-backlog tail-latency

Concurrencyproducer-consumer concurrent-backpressure

Domains that do not exist yet

— Distributed Systems — delivery guarantees, ordering scopes and what a broker can honestly promise across a partition.
— Testing & Reliability Engineering — how to test a flow whose steps live in different processes and complete at different times.