The question this answers
Three teams need to know when an order is placed. How do I tell all of them without the order service knowing they exist?
Every active subscription to a topic receives its own copy of each message published after the subscription was created, at least once. There is no atomicity across subscriptions: a message may be successfully delivered to two subscribers and permanently fail for a third, with no mechanism that notices the asymmetry.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The publisher knows the broker accepted the event. It does not know how many subscriptions exist, whether any are active, whether any are hours behind, or whether any have been failing every message for a week. A subscriber knows only its own stream; it cannot tell whether a sibling subscriber saw the same event, or in the same order.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The inversion: the producer stops naming its consumers
In a direct call or a work queue, the producer names the destination. In pub/sub it names a topic — a fact about the world, not a recipient. "OrderPlaced" is an announcement; "SendConfirmationEmail" is an instruction. The distinction is not stylistic. Topics named after instructions inevitably grow producer-side knowledge of consumers, and you have rebuilt a work queue with more machinery.
The payoff is organisational as much as technical. When the fraud team wants to score orders, they create a subscription. The orders team does not deploy, does not review, and often does not find out. That is the property you bought, and it is also the property that makes pub/sub hard to operate: you cannot enumerate the consequences of publishing an event by reading the producer’s code.
A useful discipline: an event describes something that already happened, in the past tense, with enough context to be useful without a callback. If a subscriber must call back to the producer to understand the event, the fan-out has bought you nothing — you have coupled availability again through the back door.
Fan-out means partial failure is the normal state
With three subscribers, "the event was processed" is not a proposition with a truth value. Each subscription succeeds or fails independently, at its own lag, with its own retry policy and its own dead-letter destination. The system is almost never in a state where all subscribers have caught up — it is continuously reconverging.
This is the distributed-systems reason pub/sub feels harder than it looks. The producer’s success metric is a lie by construction: it reports on the publish, which is the easy part. The genuinely load-bearing metrics live on the subscriptions, and there is one set per subscription. A new subscriber added by another team is a new set of metrics and a new page rotation that the producing team never signed up for.
Consequently, an invariant that spans subscribers cannot be maintained by the messaging layer. "Every order is either fraud-scored or held" is not something pub/sub can give you. If you need it, you need Reconciliation Is a Component, Not a Cleanup Script — a periodic sweep that compares source of truth against each derived view and repairs the difference. Treat it as part of the design, not as an incident response.
- Each subscription has its own backlog, its own lag, its own DLQ, and its own owner. If any of those is unassigned, that subscription is a future silent-loss incident.
- Retries in one subscription do not affect the others; a subscriber that retries aggressively can take down a shared downstream that another subscriber also depends on.
- Subscribers may see events in different orders relative to each other, and there is no cross-subscription snapshot at which the system is consistent.
- Deleting a subscription is a data-loss operation for that consumer; recreating it typically starts from "now", so the gap is permanent unless the topic retains history.
Filtering, and the temptation to put logic in the broker
Most brokers let a subscription filter — by routing key, by attribute, by pattern. Used lightly this is excellent: it saves a subscriber from receiving and discarding 99% of a firehose. Used heavily it becomes business logic in a component with no tests, no code review workflow, and no local development story.
The practical rule is that a filter may express which category of event I care about, and should not express whether this particular event needs action. type = "OrderPlaced" AND region = "eu" is routing. amount > 500 AND customer.tier != "gold" AND NOT flagged is a fraud rule that has escaped into infrastructure, and it will be discovered during an incident by someone reading a console.
The broker also cannot filter on anything it cannot see. If the payload is encrypted or the decision requires a database lookup, filtering must happen in the subscriber, and the fan-out cost is real: N subscribers each pulling the full firehose is N times the bandwidth and N times the deserialisation.
| Model | Copies stored | Adding a subscriber costs | Replay for a new subscriber |
|---|---|---|---|
| Broker fan-out to per-subscription queuestypical | One per subscription | Storage + delivery, linear in N | Impossible — starts from now |
| Shared log, per-group offsetsprotocol | One, shared | One more offset pointer, near zero | Free — read from any retained offset |
| Producer-side fan-out (direct calls to each)typical | None | A producer deploy | Impossible |
| Webhook fan-out to external subscriberstypical | One delivery record per endpoint | Endpoint registration + retry budget | Only if you retain the delivery log |
Ordering across a fan-out is weaker than you think
Even where a broker promises ordering, it promises it within a scope: a partition, a routing key, a single subscription’s stream. Two events published in order can reach one subscriber in order and another subscriber out of order, because the two subscriptions are independent streams with independent retries.
Retries alone break order within a single subscription. If OrderPlaced fails once and is retried while OrderShipped succeeds immediately, the subscriber processes shipped-before-placed. A handler that assumes causal order will throw, and the retry of *that* failure now interleaves with further events. This is the mechanism behind most "impossible state" bugs in event-driven systems.
The robust designs do not fight this. They make handlers order-insensitive: carry the full state in the event so a later event supersedes an earlier one; version the entity and ignore stale versions; or make the handler an upsert of a derived view rather than a delta. API Design’s webhook-ordering covers the same problem at the endpoint boundary, and the conclusion is identical.
Key points
- Publishers name a topic, not a recipient; each subscription receives its own copy with its own lag, retries and dead-letter path.
- There is no atomicity across subscriptions — success for two and permanent failure for a third is a normal, unobserved state.
- The publisher’s success metric measures only the publish. The load-bearing metrics are per-subscription and belong to other teams.
- Order is only guaranteed within a scope, and retries break it even there. Design handlers to be order-insensitive rather than assuming order.
- Events should be past-tense facts carrying enough context to act on; instructions in topic clothing recouple producer to consumer.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A subscriber registers a subscription against a topic, optionally with a filter. The subscription is durable state on the broker.
- • A publisher writes a message to the topic and receives a single acknowledgement.
- • The broker materialises the message into each matching subscription — either as a physical copy per subscription, or as a shared record with per-subscription position.
- • Each subscription is then consumed independently, with its own acknowledgement, retry and dead-letter behaviour.
- • A subscription created after a publish does not receive that message, unless the underlying store retains history and the subscription is positioned in the past.
- • A subscription exists but its consumer has been dead for days; the broker keeps accumulating copies until retention or quota is hit.
- • A filter is subtly wrong and silently excludes a class of events — no error is generated for a message that matched nothing.
- • One subscriber’s retry storm saturates a downstream shared with another subscriber.
- • A message is delivered to two subscriptions and permanently dead-lettered in a third, leaving a cross-system invariant broken.
- • A schema change breaks one subscriber and not others; the producer sees a fully successful deploy.
- • The invisible consumer: a subscription’s DLQ has 40,000 messages and nobody has an alert on it, because the team that created the subscription reorganised. The operator discovers it when a quarterly report is wrong.
- • Quota exhaustion from an abandoned subscription: a topic stops accepting publishes because one dead subscription accumulated unbounded copies. The symptom is a publish failure in a service that has nothing to do with the abandoned consumer.
- • Silent filter drop: after a routing-key change, one subscriber receives zero messages. There is no error anywhere — a filter that matches nothing looks exactly like an idle topic on every dashboard.
- • Impossible state from reordering: the operator sees a "shipment created for unknown order" error spike in one subscriber only, correlated with a transient failure in the upstream that triggered retries.
- • Fan-out amplification: a topic at 5k messages/sec with 12 subscriptions is a 60k messages/sec delivery load; adding the thirteenth subscriber is the change that pushes the broker over, and it was made by a team with no visibility into the other twelve.
- • None between subscribers, by design. That is what makes independent teams possible and what makes cross-subscriber invariants impossible.
- • The broker coordinates the subscription registry: which subscriptions exist and where each one is. That registry is durable state, and losing it loses positions, not messages.
- • Any invariant spanning two subscribers requires coordination the messaging layer does not provide — either a saga with explicit orchestration (Orchestration: One Component Owns the Workflow) or periodic reconciliation.
- • A publish acknowledged by the broker will be delivered to every subscription that existed at publish time, at least once, subject to retention.
- • Subscriptions that are down accumulate rather than lose, until retention or a quota is reached — after which loss is silent.
- • Cross-subscription consistency is not preserved and is never restored by the messaging layer itself.
- • Detect: per-subscription backlog age and per-subscription DLQ depth, with an owner attached to each. A subscription with no alert owner should fail an audit.
- • Contain: pause or delete a runaway subscription before it exhausts topic-level quota and takes publishing down with it.
- • Recover: drain each subscription independently; they do not need to recover together and forcing them to is a mistake.
- • Reconcile: compare each derived view against the source of truth and repair. This is the only mechanism that fixes a permanently dead-lettered event.
- • Verify: check every subscription’s lag returned to baseline, not just the ones you were paged about.
- • Backlog age and DLQ depth per subscription, dimensioned by subscription name so a new subscription appears on dashboards automatically.
- • A count of subscriptions per topic, alerted on growth — fan-out amplification is a capacity change made by someone else.
- • Messages matching zero subscriptions, if the broker exposes it; otherwise a canary subscriber that matches everything.
- • Per-subscription delivery-attempt distribution — a shifting distribution means a subscriber is degrading before it starts dead-lettering.
- • Reconciliation delta per derived view: the only metric that catches a permanently failed subscriber.
- • Several independent consumers need the same fact, and the set of consumers changes without the producer’s involvement.
- • The producing team must not be responsible for downstream business logic or its failures.
- • Consumers have genuinely different latency and reliability requirements for the same event.
- • Exactly one consumer exists and always will. You have added a topic, a subscription and a fan-out for no benefit — a work queue or a direct call is simpler.
- • A cross-consumer invariant must hold. Pub/sub cannot express it; you will discover this after shipping.
- • The event is really a command with one correct handler. Naming it as an event obscures the fact that a failure means the work did not happen.
- • Subscriber count is high and payloads are large — fan-out multiplies bandwidth and storage linearly, and the cost lands on the producing team’s budget.
- • A shared The Log Is Not a Queue with consumer groups: the same fan-out with one stored copy, plus replay for new consumers. Heavier to operate, dramatically better for adding subscribers later.
- • Producer-side direct calls to a known, small, stable set of consumers. Simpler and loudly failing when the set genuinely does not change.
- • Webhooks with a delivery log, when the subscribers are external. API Design owns the endpoint contract; you own the delivery guarantees.
- • Consumers polling a queryable source of truth on an interval — higher latency, no fan-out cost, and no ordering problem at all.
One event, many independent readers
What people believe, and what is true
Pub/sub guarantees all subscribers see the same events.
It guarantees each subscription receives a copy. Permanent failure in one and success in another is the ordinary case, and nothing reports the asymmetry.
The publish succeeded, so the event was handled.
The publish succeeded means the broker has the bytes. Handling is a property of N independent subscribers whose health the publisher cannot see.
Adding a subscriber is free.
In broker-fan-out models it adds a copy per message: storage, bandwidth, and delivery load, all charged to the topic. The thirteenth subscriber is a capacity decision.
We can replay the topic for the new team.
Only if the underlying store retains history and supports positioning in the past. Classic per-subscription-queue brokers cannot; a log can. Check before promising.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Publishers announce facts to a topic; every subscription gets its own copy and its own fate. Adding a consumer needs no producer change — and no component tracks whether all consumers succeeded.
Practical
Name topics after past-tense facts. Give every subscription an owner, a backlog-age alert and a DLQ alert. Make handlers order-insensitive and idempotent. Build reconciliation for any invariant that spans two subscribers, because the messaging layer will never give you one.
Advanced
Pub/sub is a deliberate refusal of a global commit point. The publisher commits one fact; the consequences of that fact commit independently, at unbounded and unequal delay, with independent failure. That is precisely the trade Eventual Consistency: If Updates Stop, Replicas Converge describes, applied to derived work rather than to replicas — and it means the reachable states of the whole system include every combination of "subscriber i has caught up". If any of those combinations is unacceptable to the business, you need coordination that pub/sub does not have, and pretending otherwise just relocates the failure to an incident.
Apply it
- 🔧 Add a subscription with a filter that matches nothing, and find every dashboard on which that is distinguishable from a healthy idle subscription. There usually is none — then build one.
- 🔧 Force a retry in one subscriber and demonstrate the resulting reordering breaking a naive handler; then rewrite the handler to be order-insensitive.
- ⚡ Publishing to a topic starts failing. The cause is an abandoned subscription created eight months ago by a team that no longer exists. Design the guardrail.
- ⚡ Your fraud subscriber has been dead-lettering every message for a week and revenue is unaffected — until it is not. What alert should have existed, and who owns it?
- 💬 Three subscribers, one event. Two succeed, one dead-letters permanently. Which component notices?
- 💬 A team asks you to replay six months of events into a new subscriber. What do you need to know before answering?
- 💬 Your subscriber occasionally gets "shipment for unknown order". No messages are lost. Explain it.