Circuit Breaker
A circuit breaker watches the failure rate of calls to one dependency and, once it is clearly down, fails fast instead of spending a timeout on every request; Closed → Open → Half-Open → Closed is the state machine, and its thresholds decide whether it protects the system or trips on noise.
When a dependency is down, every call still waits the full timeout before failing, so the caller’s threads, connections and latency budget are consumed by requests that cannot succeed; the breaker converts a slow failure into a fast one and gives the dependency room to recover.
Why failing fast is a feature
Suppose Service B is down and Service A calls it with a 2 s timeout. Every request to A that touches B now takes at least 2 s and then fails. If A handles 500 requests per second with a pool of 200 worker threads, the pool is exhausted in under half a second, and A is now also down — for *all* of its endpoints. Worse, A is still sending 500 requests per second into B, which is exactly what B does not need while it restarts. The breaker breaks this loop. After enough failures it opens: calls to B return immediately with a typed error, A’s threads stay free, and B receives no traffic except an occasional probe. When the probe succeeds, the breaker closes and normal traffic resumes.
The electrical metaphor is precise: the breaker does not fix the fault; it prevents the fault from burning down the house, and someone must still repair the wiring. A breaker that opens is a paging event, not a resolution.
The state machine and its thresholds
Closed is normal: calls pass through and each outcome is recorded in a sliding window (the last N calls, or the last T seconds). When the failure rate over that window crosses a threshold — say 50% — *and* the window holds at least a minimum number of calls — say 20 — the breaker moves to Open. The minimum matters: without it, the first two calls of the morning both failing would open the breaker for everyone. In Open, every call fails immediately without touching the dependency, for an open duration — typically 10–60 s, ideally with jitter so a fleet of instances does not probe in unison. When the open duration elapses, the breaker moves to Half-Open and lets a small number of probe calls through. If they succeed, it closes; if any fails, it re-opens and the open duration often grows (exponentially, capped).
Slow calls should count as failures. A dependency answering every request in 1.9 s against a 2 s timeout is technically succeeding and practically down; most implementations therefore also track a slow-call rate with its own threshold.
1export class CircuitBreaker {2 private state: 'closed' | 'open' | 'half-open' = 'closed'3 private results: boolean[] = [] // sliding window of recent outcomes4 private openedAt = 05 constructor(private readonly o = { window: 20, minCalls: 10, failureRate: 0.5, openMs: 30_000 }) {}6 7 async call<T>(fn: () => Promise<T>): Promise<T> {8 if (this.state === 'open') {9 if (Date.now() - this.openedAt < this.o.openMs) throw new Error('circuit open')10 this.state = 'half-open' // let exactly one probe through11 }12 try {13 const out = await fn()14 this.record(true)15 if (this.state === 'half-open') { this.state = 'closed'; this.results = [] }16 return out17 } catch (err) {18 this.record(false)19 if (this.state === 'half-open' || this.failing()) { this.state = 'open'; this.openedAt = Date.now() }20 throw err21 }22 }23 private record(ok: boolean) { this.results.push(ok); if (this.results.length > this.o.window) this.results.shift() }24 private failing() { const n = this.results.length; return n >= this.o.minCalls && this.results.filter((r) => !r).length / n >= this.o.failureRate }25}When retries make the outage worse
The breaker is the antidote to retry amplification, and the arithmetic of amplification is worth internalising. A user clicks "Pay". The API gateway retries failed upstream calls 3 times. The order service retries its payment client 3 times. The payment client library retries HTTP errors 3 times. One click becomes up to 3 × 3 × 3 = 27 requests to the payment provider. The provider was slow because it was near capacity; it is now receiving 27× its normal load from your system alone, and every other customer of that provider is doing the same thing. Latency rises, more calls time out, more retries fire. The provider’s recovery is delayed by the very clients waiting for it.
A breaker at the order service’s payment client cuts this at the second layer: after 20 calls with a 50% failure rate, the circuit opens, the 3 × 3 inner attempts stop, and the gateway’s retries hit a fast "circuit open" error instead of a 2 s timeout. The provider now sees only probes. The combination that actually works is: retry at one layer with a budget, break per dependency, and never retry across an open breaker.
1 click
└─ gateway: 3 attempts (×3)
└─ order svc: 3 attempts each (×9) ◄── breaker opens here after ~20 failures
└─ client: 3 attempts each (×27) — inner attempts stop; outer retries fail fastPer-dependency breakers, and what to return when open
One breaker per dependency — not one per service, and not one for all outbound calls. The payment provider and the recommendation service fail independently; a shared breaker would take recommendations down when payments fail, or worse, keep payments open because recommendations are healthy enough to dilute the failure rate. Some teams go finer: one breaker per dependency *and endpoint*, because GET /prices can be fine while POST /charge is timing out. The right granularity is the unit that fails together.
What the caller does when the circuit is open is a product decision, made per call site, and it is where the breaker connects to the rest of the reliability toolkit.
| Response | When it is right | Cost |
|---|---|---|
Fail fast with a typed error (503 + Retry-After) | The operation cannot be substituted: a charge, a login, a transfer | The user sees an error — honestly and in 5 ms rather than 2 s |
| Serve a cached or last-known-good value | Reads where staleness is acceptable: product details, exchange rates, feature flags | Stale data must be labelled and its age bounded; see Caching Architecture |
| Return a default / empty result | Optional features: recommendations, related items, badges | Invisible degradation; must be counted and alerted on |
| Defer to a queue and respond 202 | Work that can complete later: sending an email, charging a saved card, syncing to a CRM | Requires idempotent handlers and a status the user can poll; see Background Jobs and Workers |
Tuning and observing
The three knobs are the window (calls or seconds), the failure-rate threshold with its minimum-call floor, and the open duration. Start from the dependency’s traffic: a window of 20–100 calls and a minimum of 10–20 is sensible for a dependency receiving tens of calls per second; for one receiving a call a minute, a count window of 20 spans 20 minutes and a time window is better. The failure threshold is usually 50%; lower values trip on partial degradation (one bad instance behind a load balancer causes ~33% failures with three instances), which may or may not be what you want. The open duration should exceed the dependency’s typical restart time, and half-open probes should be few — one to five — so a recovering dependency is not knocked over by the probe wave from 50 instances.
Export the state as a metric (breaker_state{dep="payments"}), the transitions as events, and alert on open. Then the breaker is also an early-warning system: it opens seconds after a dependency degrades, long before the SLO burn-rate alert from Availability, SLOs and Error Budgets fires.
Key points
- A down dependency plus a timeout equals a caller whose thread pool fills with requests that cannot succeed; the breaker fails fast instead.
- Closed → Open on failure rate over a window with a minimum-call floor; Open → Half-Open after a timed duration; a probe success closes it, a probe failure re-opens it.
- Retry amplification: 3 retries at 3 layers is 27 attempts per request against a dependency that is slow because it is overloaded. Break per dependency; retry at one layer with a budget.
- When open, choose per call site: fail fast, serve cached, return a default, or defer to a queue — and make every option visible in metrics.
- Count slow calls as failures; a dependency answering just under the timeout is practically down.
Closed → Open → Half-Open
How data moves through it
One request or event, hop by hop.
- 1Service A → breaker: checks state before the call; open → immediate typed failure to the fallback path.
- 2Breaker → Service B: closed or half-open → the call proceeds with its own per-attempt timeout.
- 3Service B → breaker: outcome (success, error, or slow success) appended to the sliding window; threshold evaluated.
- 4Breaker → metrics: state gauge and transition events emitted; an alert fires on open.
- 5Service A → fallback: cached value, default, deferral to a queue, or an honest 503 with
Retry-Afterequal to the remaining open duration.
When to use — and when not
- Every synchronous call to an external provider or another service whose failure would otherwise consume the caller’s pool.
- Dependencies with a known recovery behaviour (restart, failover) that need traffic to back off for them to recover.
- Alongside a fallback that the product has explicitly approved for that call site.
- Very low-traffic dependencies where the window never reaches the minimum-call floor: the breaker will never open, and a plain timeout does the job.
- In-process calls or local disk: nothing to break, and the machinery adds a state to reason about.
- As a substitute for fixing a dependency that is permanently under-provisioned: an open breaker is a symptom, not a capacity plan.
Tradeoffs
Small code, but three thresholds per dependency that must be tuned to real traffic; the state machine is easy, the thresholds are the work.
How it fails
- Too-low minimum calls: two failures at 6 a.m. open the circuit for all traffic, and the "outage" is the breaker itself.
- One breaker shared across dependencies: a healthy dependency’s successes hide a failing one, or a failing one blocks a healthy one.
- All instances probing at once when the open duration elapses without jitter: the recovering dependency is hit by a synchronised wave.
- Open circuit swallowed as a silent fallback: the breaker has been open for hours and nobody knows because the fallback returns 200.
- Breaker inside the retry loop: each retry re-checks and re-trips the breaker, and the amplification it was meant to stop continues.
How it scales
- Breaker state is per instance; 50 instances each discover the outage independently, which costs about 50 × minCalls extra failed calls and is usually acceptable.
- Shared breaker state (in Redis) gives fleet-wide reaction in one window but adds a dependency to the path that protects against dependencies; rarely worth it.
- As dependencies multiply, breakers should be created from a registry with per-dependency config, not hand-wired; a service mesh sidecar can host them uniformly.
How it interacts with databases, queues, caches, APIs and external systems
- External APIs: one breaker per provider (and often per endpoint); the open duration is set from the provider’s documented failover time.
- Databases: a breaker around a replica lets reads fail over to the primary when the replica is unreachable; see Replication and Read Scaling.
- Caches: the usual fallback for read paths when the circuit to the source is open; label the data as stale.
- Queues: the usual fallback for write paths — enqueue the work, return 202, and let a worker retry when the circuit closes.
- Service mesh: Envoy-style sidecars implement outlier detection (per-host ejection) and breakers uniformly, moving the pattern out of application code; see Service Discovery.