Distributed Systems Cheat Sheet

The problem says X → think Y, with the reason attached. Every right-hand column is a thought, never a product: “queue keeps growing → Kafka” is the reflex this domain exists to replace, and “your consumers are slower than your producers, so capacity, admission or priority must give” is something you can act on.

The problem says	Think	Why	Lessons
Remote call timed out	The outcome is unknown, not failed	Three of the five realities behind a timeout involve the work having completed. Treating it as failure is wrong more often than it is right.	A Timeout Tells You Nothing About Whether It Happened Partial Failure: The Founding Condition The Retry Is a Decision, Not a Reflex
A retry might duplicate the operation	Idempotency	Retries are unavoidable, so duplicates are the tax. Make the second attempt harmless rather than trying to prevent it.	Idempotent Is a Property of the Whole Effect, Not the Write What Counts as the Same Operation?Deduplication: Bounded Memory Against an Unbounded Stream
Need exactly one active owner	Lease plus fencing token	Election alone does not stop a paused old owner from acting. The resource must reject the stale token.	Leader Election: Choosing One, and Knowing You Were Chosen Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely The Stale Lock Holder: A Paused Process Does Not Know It Was Paused
Two replicas disagree	Conflict resolution, and a rule chosen in advance	Divergence is normal after a partition. What matters is whether your merge rule loses data silently.	Two Writes, No Order, One Answer Required Last Write Wins Is Data Loss You Chose by Default Version Vectors: Making the Conflict Visible CRDTs: Deterministic Merge, Not Correct Merge
Everyone must see the same order	Total order broadcast — which means consensus	Total ordering and consensus are equivalent problems. That equivalence is why it is expensive.	Total Order Broadcast Is Consensus Wearing a Different Hat Four Orderings, Four Prices What Consensus Actually Solves
Need writes to stay available under partition	Weaker coordination, plus reconciliation	You are choosing the A side of CAP for this operation. Decide what repairs the invariant afterwards.	CAP: What the Theorem Actually Says Eventual Consistency: If Updates Stop, Replicas Converge Reconciliation Is a Component, Not a Cleanup Script
Need more read throughput	Replicas, with an explicit staleness policy	Read scaling is cheap. The cost is that every read site now needs a decision about staleness.	Why Replicate: What a Second Copy Buys You Read-After-Write: Letting a User See Their Own Change Monotonic Reads: Never Let Time Run Backwards
Need more write throughput or storage	Partitioning	Replication does not scale writes — every replica does every write. Only partitioning divides the work.	Why Partition: Four Ceilings, Four Different Answers Hash Partitioning and the Modulo Trap The Ring: Keeping the Mapping Stable When Membership Changes
One key gets enormous traffic	Hot key — and more partitions will not help	A single key lands on a single partition regardless of hash quality. Cache, split, replicate or coalesce.	Hot Partitions: The Skew Hashing Cannot Fix Sharding Does Not Help a Single Key One Key Expires and Five Hundred Instances Miss at the Same Millisecond
Queue depth keeps growing	Backpressure — consumers are slower than producers	An unbounded queue converts overload into latency collapse. Bound it and decide what to shed.	Backpressure Is a Signal That Has to Travel — and Reach Someone Who Can Slow Down Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help Decide at the Door Whether the Capacity Exists
One dependency's outage takes down everything	Containment: bulkheads, breakers, timeouts	Without isolation, a slow dependency consumes the caller's threads and the failure travels upstream.	Cascading Failure: When the Response to Failure Causes More Failure Containment Is Decided by What Is Shared, Not by Where the Service Boundaries Are Bulkheads: Buying Independence by Giving Up Utilisation
Retry traffic is larger than real traffic	Backoff, jitter and a retry budget	Retries at three tiers with three attempts each is 27x amplification. Cap retries as a fraction of original load.	One Retry per Tier Is Not One Retry — It Multiplies Cap Retries as a Fraction of Traffic, Not as a Count per Request Without Jitter, Every Client That Failed Together Retries Together
A change must span two services	Saga, or move it into one transaction	Ask first whether the boundary is right. A cross-service transaction is often a misplaced service boundary.	Atomicity Stops at the Process Boundary Sagas: Trading Isolation for Availability Four Questions That Test a Proposed Boundary
A message may arrive twice	Idempotent consumer	At-least-once is what a network can offer. The consumer, not the broker, is where exactly-once effects are achieved.	Where You Put the Acknowledgement Decides Everything Exactly-Once Is a Scope, Not a Guarantee Acknowledgement: The Two-Line Protocol That Decides Your Delivery Semantics
Several systems need the same event	Pub/sub, or a log if replay matters	If a consumer added next year must see history, you need retention — a queue cannot provide it.	Publish/Subscribe: One Event, Many Independent Readers Queue or Pub/Sub: Answer the Question in One Sentence The Log Is Not a Queue
Need to reprocess history	A durable, retained event log	Replay is a property of retention plus consumer-tracked offsets, not of any particular product.	The Log Is Not a Queue A Topic Is Not One Log: Ordering Lives Inside a Partition Commit Before or After: There Is No Third Option
Need ordering for a given entity	Partition by that entity's key	Ordering is guaranteed within a partition. Keying by entity is how you buy per-entity ordering.	A Topic Is Not One Log: Ordering Lives Inside a Partition Hash Partitioning and the Modulo Trap Four Orderings, Four Prices
Need something globally unique	Coordination, or partitioned ownership	Either one node decides, or the decision is serialized. There is no third option that is also cheap.	Distributed Uniqueness: One Name, Many Shards Coordination Couples Availability Start From the Invariant, Not From the Architecture
Writes in two regions conflict	A write model chosen deliberately	Single-writer, multi-writer and partitioned ownership have very different conflict and latency profiles.	Three Ways to Accept a Write in More Than One Place Active-Active: Every Conflict Scenario Becomes Real Two Writes, No Order, One Answer Required
All replicas are in one zone	Correlated failure — you have one failure domain	Three replicas that fail together are one replica with extra cost. Independence is the property, not the count.	Correlated Failure: The Independence Assumption Is Usually False Fault Domains: What Fails Together Redundancy Is Not Resilience
Log timestamps from two nodes disagree	Causal ordering, not wall-clock sorting	Clock skew means you cannot sort a distributed log by timestamp. Correlate by trace id instead.	Clock Skew: The Gap You Cannot Measure From Inside Happens-Before: The Only Ordering You Actually Have Three Nodes, Three Logs, and You Cannot Sort by Timestamp
A user does not see their own write	Read-your-writes	The narrow guarantee fixes the visible bug far more cheaply than global consistency would.	Read-After-Write: Letting a User See Their Own Change Session Guarantees: The Underrated Middle Ground Asynchronous Replication: The Loss Window You Chose
Fan-out request is slow but each shard is fast	Tail latency at scale	With 100 shards each 1% slow, the chance at least one is slow is 63%. Component tail becomes aggregate median.	Fan Out to 100 and the Component’s Tail Becomes the System’s Median Send a Second Request After p95 and Take Whichever Answers First One Slow Task Sets the Pace for Everything
The search index disagrees with the database	Derived state needs reconciliation	Decide which is authoritative, then build the repair path. Drift is expected, not exceptional.	Source of Truth: The Question Every Inconsistency Incident Is Really Asking Materialized Views: A Read Model That Lags Reconciliation Is a Component, Not a Cleanup Script
Services must all deploy together	Distributed monolith	You are paying every distributed cost and getting no autonomy. The boundary is in the wrong place.	The Distributed Monolith: All of the Cost, None of the Autonomy Four Questions That Test a Proposed Boundary The Shared Database: An Honest Trade, Not a Prohibition
An agent retried and charged twice	Agent idempotency	A model retrying an unacknowledged tool call is the timeout-ambiguity case. Side-effecting tools need keys.	The Model Retries Because It Cannot See the Result The Failures That Produce No Errors Idempotent Is a Property of the Whole Effect, Not the Write

26 of 26 rows shown. A row is a starting thought, not a design. Every one of them ends in the same two questions: what guarantee does this need, stated precisely enough to test — and what does each node actually know when it decides?