The question this answers
Which faults can I actually inject, and which of them will tell me something I do not already know?
An injected fault guarantees only that the system experienced *that* fault, at *that* injection point, at *that* radius. It does not guarantee the system experienced the real-world failure the fault is a model of — an injected 500 is not a crashed process, and a dropped link is not a partition.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node cannot distinguish an injected fault from a real one, and that is required for the test to mean anything. What matters epistemically is the reverse direction: the *experimenter* frequently cannot tell whether the fault landed. A latency injection applied at a proxy the caller does not use, or a crash of an instance already out of rotation, produces a clean run and a false confirmation. Evidence that the fault applied must come from the target’s own signals.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The catalogue
There are seven faults worth having in a programme. They differ in what they model, in how hard they are to produce faithfully, and — the important axis — in how likely they are to refute something you believe.
Read the last column first. The faults that are trivial to inject mostly exercise error-handling code you wrote deliberately, so they confirm rather than surprise. The faults that are hard to inject are hard precisely because they violate assumptions built into your infrastructure, which is the same reason they violate assumptions built into your software.
| Models | How hard to inject faithfully | What it tends to reveal | |
|---|---|---|---|
| Added latencytypical | A slow dependency, GC, cold cache | Easy — proxy, sidecar or client hook | Timeout budgets, pool occupancy, and whether latency becomes capacity loss |
| Dependency errortypical | A dependency returning 5xx | Easy — return a synthetic status | Fallback paths that have never executed; error classification bugs |
| Node crashtypical | Instance loss, OOM kill, spot reclaim | Easy — terminate the instance | Failover time, in-flight request handling, session assumptions |
| Packet losstypical | A degraded link | Moderate — needs traffic control at the host | Retransmit behaviour, tail latency, and health checks that flap |
| Disk full / IO errortypical | Exhausted volume, failing disk | Moderate — fill a volume or use a fault-injecting filesystem | Write paths with no error handling; logs that block the request path |
| Network partitionassumption | A split cluster where both halves live | Hard — needs symmetric, sustained, topology-aware blocking | Split brain, stale leaders, quorum assumptions, dual writes |
| Clock skewassumption | Drifting or jumping clocks across nodes | Hard — usually needs host-level or virtualised clock control | Expiry and lease logic, log correlation, LWW conflict resolution |
Why partitions are hard, and why they matter most
Dropping a link is not a partition. A partition is a *sustained, symmetric* loss of connectivity in which both sides remain alive and continue serving, each believing the other is gone. That combination is what produces split brain, stale leaders, dual writes and divergent state — and reproducing it takes more than a firewall rule.
The specific difficulties. It must be symmetric: an asymmetric block, where A cannot reach B but B can reach A, is a different and rarer fault, and testing it by accident while believing you tested a partition is worse than not testing. It must be sustained past every failure detector’s threshold, or you have tested a blip. It must be topology-aware: blocking service-to-service traffic while leaving the shared database, the control plane and the service mesh reachable is not a partition, it is a routing change. And it must not sever the observability path, or you cannot see what either side believed.
The payoff is that partitions test the claims with the worst consequences when wrong. Does the old leader keep accepting writes? Do both sides accept conflicting updates? Does a lease expire on the side that lost the lock manager? Does the fencing token actually fence? These are [[split-brain]], [[fencing-tokens]] and [[stale-lock-holders]] moving from theory to evidence, and no easier fault gets near them.
- Node 1 — still accepting writes — has not noticed
- Node 3 — elected by the majority side
- n1believes “I am the leader in term 7”✕ and it is false
- n3believes “I am the leader in term 8”✓ and it is true
- n2believes “the cluster is healthy”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Why clock skew is hard, and what it breaks
Clock skew is the other high-value, high-difficulty fault, and it is harder than partitions in one respect: most environments actively resist it. NTP will correct your drift while you are trying to measure its effects. Container clocks are usually the host’s clock and cannot be moved per-container. Managed platforms often do not expose the knob at all. Achieving faithful skew usually means a dedicated VM, a virtualised clock, or a time-shim library linked into the process.
It is worth the trouble because clock assumptions are buried in code nobody thinks of as time-dependent. Token and certificate expiry compared against local time. Cache TTLs computed on one node and evaluated on another. Lease and lock expiry, where a fast clock releases early and a slow clock holds past its grant. Last-write-wins conflict resolution, where a node with a fast clock silently wins every conflict — permanently, and with no error. Scheduled jobs that run twice or not at all. Log correlation, where skew makes effects appear before their causes.
And note the pattern: nearly all of those fail *silently*. A partition produces errors somebody can see. Skew produces wrong answers that look right, which is why [[clock-skew]] and [[monotonic-vs-wall-clock]] are the theory this fault is the practical test of.
injection: node-b wall clock +45s relative to fleet (NTP disabled for window)
t+00:03 node-b issues lease expires_at=12:00:45 (fleet time 11:59:15)
t+00:03 node-a sees lease valid until 12:00:45 -> holds off for 90s
t+01:10 node-b LWW write timestamp 12:01:52 beats node-a's 12:01:08
-> node-a's LATER write silently discarded
t+02:40 node-b emits log line at 12:03:22; node-a's causally
later line reads 12:02:39 -> effect precedes cause in the log
t+04:00 cache entry written by node-b, TTL 60s, treated as fresh
by node-a for 105s
errors raised during the window: 0
alerts fired: 0Injection points change what you are testing
Where you inject decides which layer is under test, and the same nominal fault at two points tests different things. Injecting latency in the *client library* tests your timeout handling but leaves the network, the pool and the sidecar untouched. Injecting at a *proxy or service mesh* tests everything below the application. Injecting at the *host* with traffic control tests the real network stack, including retransmits and connection behaviour. Injecting at the *infrastructure* level — terminating the instance, detaching the volume, revoking the credential — tests the platform’s response as well as yours.
Choose the point furthest down the stack that your hypothesis needs, because every layer you skip is a layer you have assumed away. A hypothesis about timeout budgets is fine at the client. A hypothesis about surviving instance loss is meaningless anywhere but at the infrastructure.
One rule regardless of point: prove the fault landed. The most common defective experiment is a clean run against a target that was not receiving traffic, and its output is a false confirmation, which is worse than no experiment because it retires the question. The evidence must come from the target — a latency histogram that shifted, a connection counter that dropped to zero — not from the runner reporting that it applied the rule.
- Client library: cheapest, tests your handling code only.
- Sidecar or mesh: tests the application end to end without touching the host network.
- Host traffic control: real network behaviour — retransmits, RTT, connection resets.
- Infrastructure API: instance termination, volume detach, credential revocation — tests the platform too.
- In all cases: evidence of landing comes from the target, never from the injector.
Key points
- The catalogue: latency, dependency error, node crash, packet loss, disk full, network partition, clock skew.
- Easy faults mostly confirm the error handling you wrote deliberately; hard faults refute assumptions you did not know you had.
- A partition requires both halves alive and serving, symmetric and sustained — a dropped link is not a partition.
- Clock skew is resisted by NTP and by container platforms, and nearly everything it breaks fails silently.
- The injection point determines which layers are under test; anything above it is assumed away.
- Always prove the fault landed, from the target’s signals rather than the injector’s intent.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Pick the fault that models the failure your hypothesis is about — not the one that is easiest to produce.
- • Pick the injection point furthest down the stack that the hypothesis requires.
- • Confirm the target is actually receiving traffic before injecting.
- • Apply the fault at the declared radius, for the declared duration, with the abort condition armed.
- • Verify from the target that the fault landed — a shifted latency distribution, a dropped connection count, a skewed clock read.
- • Observe the hypothesis metric, and revert on threshold or elapsed time.
- • The fault is applied to an instance that was not serving traffic.
- • A retry layer between the injection point and the application masks the fault entirely.
- • A "partition" is asymmetric or short-lived, so a different and much milder fault is what actually got tested.
- • NTP corrects the injected skew mid-experiment and the fault silently disappears.
- • The injection escapes its scope — a host-level rule catches traffic belonging to co-located workloads.
- • The injection cannot be reverted: a filled disk, a revoked credential or a terminated stateful node may not restore cleanly.
- • False confirmation: the operator sees a clean experiment and records the hypothesis as held, while the target’s request counter shows zero traffic for the entire window.
- • Fault masked by a retry layer: the operator sees no application-level effect from a 30% error injection, because the mesh retried every failure transparently and the application never saw one.
- • Not actually a partition: the operator sees no split brain and concludes the cluster is safe, while packet counters show the block was one-directional and the minority side kept receiving heartbeats.
- • Skew silently corrected: the operator sees the expected failures for ninety seconds and then normality, because NTP stepped the clock back mid-window — and the report says the system "recovered".
- • Unrevertable injection: the operator finds the volume still full after the experiment ended, because filling it triggered a process that cannot restart without free space.
- • Collateral scope: the operator sees an unrelated co-located workload degrade, because a host traffic-control rule matched more than the target’s traffic.
- • Host-level and infrastructure-level injections affect anything sharing the host or account, so they need coordination with whoever else is there — a scope check is part of the design, not a courtesy.
- • Clock injection usually requires disabling time synchronisation for the window, which is a platform-level change with its own blast radius.
- • The injection itself must not require the target’s cooperation: a fault the application opts into cannot model the application being gone.
- • Revert paths for stateful faults (disk, credentials, stateful nodes) need a plan agreed in advance with the component owner, because they are the ones that do not simply undo.
- • Under latency and error injection, the system provides its degraded-path guarantees — which is precisely what is being measured.
- • Under a faithful partition, each side provides only what it can guarantee alone: usually availability without agreement on one side, and agreement without full availability on the other.
- • Under clock skew, guarantees that are stated in wall-clock terms — leases, TTLs, expiry, LWW ordering — are simply not in force, and nothing reports that.
- • Under disk-full, durability claims that depend on being able to write are suspended, including the write-ahead log the recovery path needs.
- • Detect: watch the abort metrics, and separately watch evidence that the fault is still landing — a fault that stops mid-run invalidates the result.
- • Contain: revert the injection first; diagnose afterwards. A live injection during an unexpected deviation is a variable you can remove instantly.
- • Recover: for stateful faults, follow the pre-agreed restoration path — free the disk, reissue the credential, rejoin the node, re-enable time sync.
- • Reconcile: after a partition or skew injection, expect divergence. Reconcile the two sides explicitly rather than assuming convergence happened.
- • Verify: confirm the fault is fully removed — check NTP is re-enabled, traffic rules cleared, instances back in rotation — before recording the result.
- • Proof of landing from the target: latency distribution shift, error counter, connection count, or the node’s own clock read.
- • Symmetry and duration evidence for partitions: packet counters in both directions across the whole window.
- • Whether an intermediate layer absorbed the fault — compare injected fault rate against the rate the application observed.
- • Scope evidence: which workloads deviated, versus which were declared in the radius.
- • Post-experiment cleanliness: no residual traffic rules, no disabled time sync, no drained instance left drained.
- • Testing hypotheses about failure classes your system claims to survive but has not survived recently.
- • Validating a fix for a failure mode that is hard to reproduce naturally — the partition or skew that caused last quarter’s incident.
- • Exercising code paths that real traffic essentially never reaches: fallbacks, compensations, recovery branches.
- • When the fault is chosen for ease rather than relevance — a programme of latency injections can run for a year and never touch the assumptions that matter.
- • When the injection cannot be evidenced, since an unverifiable run produces a confident wrong answer.
- • Stateful faults with no tested revert, where the experiment’s worst case is not the fault but the cleanup.
- • Deterministic simulation testing: run the whole system on a simulated network and clock where partitions and skew are trivial and reproducible. Far stronger for protocol-level bugs, and it cannot test your real infrastructure.
- • Property-based or model-checked tests of the protocol logic, which find split-brain bugs without touching production at all.
- • Reproducing the fault in staging with a synthetic load, when the hypothesis is about mechanism rather than scale.
- • Waiting for the fault to occur naturally and studying it well — free, and the only reason it is not the primary method is that you do not choose the timing or the radius.
Seven faults, and the two that will actually tell you something
| Injection point | Difficulty | Value | Proof it landed | |
|---|---|---|---|---|
| Added latencytypical | proxy, sidecar, or the client library on the caller | 1/5 | 2/5 | the target’s own request-duration histogram shifts, not the injector’s intent log |
| Dependency errortypical | the dependency’s handler, or a proxy returning 500 | 1/5 | 2/5 | the caller’s per-dependency error counter for that status |
| Node / pod crashtypical | the orchestrator, or a kill signal on the host | 2/5 | 3/5 | the instance disappears from the endpoint list and the replica count moves |
| Packet lossassumption | the host network stack, or the CNI | 3/5 | 3/5 | retransmit counters on both ends, not just the injector |
| Disk fullassumption | the container or host filesystem | 3/5 | 3/5 | the target’s own free-space metric, and its write errors |
| Network partitionassumption | firewall rules on both sides, or the service mesh | 5/5 | 5/5 | packet counters on both sides showing symmetry and duration |
| Clock skewassumption | the node’s time source, with time sync disabled for the window | 5/5 | 5/5 | the node’s reported offset held for the whole window |
What people believe, and what is true
Blocking traffic between two services is a partition test.
A partition needs both halves alive and serving, symmetric, and sustained past every failure detector’s threshold. A one-way block for ten seconds tests something much milder.
We inject latency and errors regularly, so our fault coverage is good.
Those exercise the error handling you wrote on purpose. The assumptions that break are behind partitions and clock skew, which is exactly why those are the ones that never get injected.
The experiment ran cleanly, so the system handled the fault.
Or the fault never landed. Without evidence from the target — a shifted distribution, a dropped counter — a clean run is indistinguishable from no experiment.
Clock skew is not realistic; we run NTP.
NTP fails, steps, and is misconfigured; VMs pause and resume; leap seconds happen. And the reason to test is that skew failures are silent — you would not know it had occurred.
Injecting at the client library is equivalent to injecting on the network.
It skips the pool, the sidecar, the host stack and the platform. Everything you skip is a layer whose behaviour you have assumed instead of tested.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Seven faults: latency, dependency error, node crash, packet loss, disk full, partition, clock skew. The first five are easy; partitions and clock skew are hard, and they are where the real findings are.
Practical
Choose the fault your hypothesis needs and the deepest injection point it requires. Prove the fault landed from the target’s own signals. For partitions, verify symmetry and duration with packet counters. For skew, disable time sync for the window and confirm the offset held. Plan the revert for stateful faults before you inject.
Advanced
Rank faults by the gap between how much your code assumes about them and how often production exercises them. Latency and errors are exercised daily, so the code is battle-tested and the injection mostly confirms. Partition and skew are exercised approximately never, so the code embodies untested assumptions accumulated over years — which is why the difficulty of injecting them and the value of injecting them have the same cause.
Apply it
- 🔧 Take a fault your team injects routinely and name the assumption it tests. If it only tests error-handling code you wrote on purpose, pick a harder fault.
- 🔧 Design a clock-skew injection for one node of a service that uses leases, and list every mechanism the skew would silently affect.
- ⚡ A partition experiment shows no split brain. Packet counters reveal traffic flowed from the minority side to the majority side throughout. What did you actually test, and what do you change?
- 💬 What is the difference between blocking traffic between two services and injecting a network partition?
- 💬 Why is clock skew hard to inject, and what does it break that a crash does not?
- 💬 Your latency injection ran with no observable effect. List the explanations, in the order you would check them.
- 💬 You can inject at the client library, the mesh, the host or the cloud API. How do you choose?