The question this answers
When a dependency fails, which parts of my product should keep working — and did anyone decide that before today?
Per feature, not per service: for each user-visible feature, a named set of dependencies whose failure the feature survives with reduced quality, and a named set whose failure it cannot survive. The guarantee is that the first set degrades to a defined fallback rather than to an error page. It says nothing about the second set.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A request handler knows which of its calls returned, which errored and which timed out. It does not know whether the failing dependency is failing for everyone or only for this request, and it does not know whether the caller above it can tolerate a partial answer. That second question cannot be answered locally — it is a product decision that has to be encoded into the handler in advance, as a policy, because at request time there is nobody to ask.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Criticality is a property of features, not of services
The usual framing — "is this service critical?" — has no answer, and the argument it produces is unresolvable because both sides are right. The recommendation service is critical to the personalised homepage and irrelevant to checkout. The pricing service is critical to both. The fraud service is critical to checkout and irrelevant to browsing. There is no single criticality number for a service; there is a matrix of feature against dependency, and every cell holds a decision.
Filling that matrix in is the engineering content of graceful degradation. Not the try/catch, not the fallback value — those are trivial once the cell is filled. The hard part is that each cell requires someone with product authority to say what the user should see when the dependency is gone, and that person is not available at 03:00.
The test for whether the matrix exists: pick a dependency and ask an engineer *which features continue and in what state* if it disappears for an hour. If the answer takes more than a few seconds, the decision has not been made, and it will be improvised during the incident by whoever is on call.
| Recommendations | Pricing | Inventory | Fraud scoring | |
|---|---|---|---|---|
| Browse cataloguetypical | Degrade: show best-sellers | Degrade: show cached price with "as of" label | Degrade: hide stock badge | Not on path |
| Product pagetypical | Degrade: hide the carousel | Critical: cannot show a product without a price | Degrade: hide stock badge | Not on path |
| Add to carttypical | Not on path | Degrade: re-price at checkout | Degrade: allow, validate at checkout | Not on path |
| Checkoutassumption | Not on path | Critical | Critical: cannot sell what does not exist | Policy decision — see below |
The fallback has to be designed, not defaulted
Once a cell says "degrade", something concrete has to happen in the code, and the space of options is larger than "return empty". A stale cached value with an explicit staleness label is usually better than a blank; a static default is usually better than a spinner that never resolves; an honest "we could not load this right now" is usually better than silently showing zero, because zero is a number the user will believe.
The ordering rule that catches most teams: degrade before you are forced to. A handler that waits the full timeout on a dependency it has already decided is optional has spent the entire latency budget to learn something it did not need. If recommendations are optional, give them a deadline far shorter than the request deadline and move on. This is where [[timeout-budgets]] and [[deadline-propagation]] become the mechanism of degradation rather than a separate topic.
And degrade *fast* on repeat: once a dependency has failed consistently, continuing to call it on every request converts an optional dependency into a latency tax on every user. Architecture owns the pattern that solves this — link circuit-breaker, do not reimplement it here.
1// The interesting artefact is this table. It is the degradation matrix,2// expressed so that it cannot drift away from the code that enforces it.3const policy = {4 recommendations: { role: 'optional', budgetMs: 80, fallback: 'bestSellers' },5 pricing: { role: 'critical', budgetMs: 400 },6 inventory: { role: 'optional', budgetMs: 120, fallback: 'hideBadge' },7} as const8 9async function buildProductPage(id: string, deadline: Deadline) {10 // critical: its failure is the page's failure, and that is a decision11 // someone made, not a default the framework chose.12 const price = await call(pricing, id, deadline.slice(policy.pricing.budgetMs))13 14 // optional: bounded by its own much smaller budget, so a slow dependency15 // cannot spend the page's latency on information we agreed to live without.16 const recs = await callOrFallback(17 recommendations, id,18 deadline.slice(policy.recommendations.budgetMs),19 policy.recommendations.fallback,20 )21 22 return render({ price, recs, degraded: recs.isFallback })23 // ^ the response says it is degraded,24 // so the caller and the dashboard know.25}Fail-open or fail-closed is the same question with consequences
One row in the matrix above was deliberately left as a policy decision: fraud scoring during checkout. If the fraud service is unavailable, do you accept the order unscored, or reject it? Accepting maximises availability and accepts fraud loss. Rejecting maximises safety and loses revenue from legitimate customers. Neither is correct in general — the correct answer depends on the cost asymmetry, and it must be decided by someone who owns that cost.
The distributed-systems content here is narrow and worth stating precisely: the choice is forced by `[[timeout-ambiguity]]`. You do not get to wait for the fraud service to come back, because "unavailable" and "slow" are indistinguishable from where the checkout handler stands, and the user is waiting. So every optional-security dependency has an implicit default, and the failure is not choosing badly — the failure is not knowing which default your code currently has.
The full treatment of the trade-off, including the cases where fail-open is a security vulnerability rather than an availability feature, belongs to Security and is linked below. Our contribution is only this: it is a cell in the degradation matrix like any other, and it must be filled in before the incident.
- Fail-open: proceed without the check. Preserves availability; accepts whatever the check was preventing.
- Fail-closed: refuse. Preserves the invariant; converts a dependency outage into a product outage.
- The unacceptable third option: whichever one your
catchblock happens to implement, which nobody has read. - Whichever you choose, emit a distinct signal — the count of requests that took the degraded path is the only way anyone learns it is happening.
Degradation is invisible unless you instrument it
A well-degraded system looks healthy. Error rate at baseline, latency fine, availability green — because degradation is precisely the act of turning an error into a successful, worse response. This is the desired behaviour and it is also a monitoring trap: the system can serve a badly degraded experience to every user for days while every SLI reports success.
The fix is to count the degraded path as its own signal, per feature and per dependency, and to alert on the *rate of degradation*, not only on errors. A useful phrasing for the alert: "more than 5% of product pages rendered without recommendations for 10 minutes" is an incident even though nothing failed.
# looks perfect, and is not
http_5xx_rate{feature="product_page"} 0.0008 OK
p99_latency_ms{feature="product_page"} 210 OK
# the signals that actually describe the user's experience
feature_degraded_ratio{feature="product_page",dep="recommendations"} 0.97
feature_degraded_ratio{feature="product_page",dep="inventory"} 0.00
fallback_served_total{dep="recommendations",kind="bestSellers"} 41_882
dependency_budget_exceeded_total{dep="recommendations"} 42_106
# reading: for the last hour essentially nobody has seen a real
# recommendation, and no error-based alert can ever say so.Key points
- Criticality is a property of a (feature, dependency) pair, not of a service.
- The engineering work is filling in the matrix in advance; the try/catch is the easy part.
- Optional dependencies need their own short deadline, or they spend the whole request budget proving they are down.
- Fail-open versus fail-closed is one cell of the matrix, forced by the fact that slow and down are indistinguishable.
- A degraded system reports healthy — degradation must be counted as its own signal or it is invisible.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • For each user-visible feature, enumerate the dependencies on its request path.
- • For each pair, record one of three verdicts: critical, degradable (with a named fallback), or not on path.
- • Encode the verdict as data next to the call site, so the policy and the code cannot drift apart.
- • Give degradable dependencies a deadline far smaller than the feature’s overall budget.
- • On breach, serve the fallback and mark the response as degraded, both to the caller and to telemetry.
- • Alert on sustained degradation ratio, separately from error rate.
- • The fallback path is itself untested and throws the first time it executes, converting a degradation into an outage.
- • The fallback reads a cache that is fed by the very dependency that is down, so it is empty exactly when it is needed.
- • A dependency marked optional is actually load-bearing for a downstream consumer of the response, which now receives a plausible-looking wrong answer.
- • The optional call is bounded by a timeout but not by a connection-pool limit, so slow optional calls still exhaust a shared resource.
- • Degradation cascades: the fallback for A calls B, and B is the reason A failed.
- • Silent quality collapse: the operator sees availability at 99.99% and a normal error rate while 97% of product pages have been rendering without recommendations for six hours, because no signal counts the fallback.
- • Latency tax on an optional dependency: the operator sees p99 on the product page jump from 200ms to 2.1s during an outage of a service the team describes as "not critical" — because the handler waits the full request deadline for it.
- • Fallback crash: the operator sees error rate rise *when the dependency starts failing*, not before — the failure is in the rarely executed fallback branch, and the stack trace names the fallback, not the dependency.
- • Unintended fail-open: the operator learns from a fraud report, not from a dashboard, that the scoring service was down for two hours and every order in that window was accepted unscored.
- • Degradation cascade: the operator sees a second service fail two minutes after the first, because the first service’s fallback path calls it at ten times the normal rate.
- • Filling the matrix requires coordination between engineering and whoever owns the product decision — that is the expensive part, and it is done once, offline.
- • At request time, degradation requires no coordination at all: each handler applies a local policy. That locality is what makes it work during a partition.
- • A shared kill switch adds coordination and is worth it: an operator flipping "recommendations off" globally is faster and more predictable than every handler independently discovering the dependency is down.
- • Coordinating the *fallback data* — a warm cache of best-sellers, a static price snapshot — is a background job, so its coordination cost is paid outside the request path.
- • Features whose dependencies are all degradable continue to serve, with defined and labelled reduced quality.
- • Features with a failed critical dependency fail, and should fail quickly and legibly rather than hanging.
- • Any invariant that the skipped dependency was enforcing is unenforced for the duration — that is the fail-open bill, and it comes due at reconcile.
- • Response payloads carry a degraded marker, so downstream consumers can make their own decision rather than inheriting yours silently.
- • Detect: alert on degradation ratio per feature and dependency, not only on errors.
- • Contain: flip the kill switch for the failing dependency so every handler takes the fast path instead of discovering the failure per request.
- • Recover: restore the dependency and re-enable it gradually — a cold dependency taking full traffic is the classic second outage.
- • Reconcile: for every fail-open decision taken during the window, re-run the skipped check offline. Unscored orders get scored; unvalidated inventory gets validated.
- • Verify: confirm the degradation ratio has returned to zero and that the reconcile pass over the fail-open window produced no unresolved items.
- • Degraded-response ratio per feature per dependency — the primary signal, and one most systems do not have.
- • Count of responses served from each named fallback, which distinguishes "fallback worked" from "fallback never fired".
- • Latency attributable to optional dependencies, so a supposedly optional call that is spending the request budget is visible.
- • Fail-open counter per security-relevant dependency, and the size of the resulting reconcile queue.
- • Fallback-path error rate, tracked separately, because it is the code least likely to have been exercised.
- • Products with a clear quality gradient — a page that is worth serving without its carousel, a feed worth serving slightly stale.
- • Systems with many optional enrichment dependencies, where the arithmetic of independent availabilities otherwise makes the whole worse than any part.
- • Anywhere the alternative is an all-or-nothing page, since a single 99.9% dependency then caps the whole feature at 99.9%.
- • Operations with a correctness invariant that the skipped dependency was enforcing: degrading a payment authorisation is not degradation, it is a defect.
- • When the degraded output is indistinguishable from the real one — a silently stale price is worse than an error, because the user acts on it.
- • Small systems where the matrix has three cells and the ceremony of maintaining it exceeds the benefit of writing the fallback inline.
- • Remove the dependency from the request path entirely: precompute the enrichment and serve it from local state, so there is nothing to degrade.
- • Make the feature asynchronous — return the page and load the optional part client-side, so its failure is scoped to one widget by construction.
- • Fail fast and honestly: for features where a partial answer is misleading, an immediate clear error beats a plausible wrong one.
- • Cache the dependency’s output aggressively with an explicit staleness bound, which converts an availability problem into a freshness problem you can reason about.
Which dependency is actually critical, per feature
| Feature | Payments 99.90% | Inventory 99.90% | Pricing 99.950% | Auth 99.990% · shared | Session store 99.90% · shared | Search index 99.50% | Reviews 99.00% | Recommendations 98.00% | Feature availability |
|---|---|---|---|---|---|---|---|---|---|
| Checkout | 97.65% 6 critical · 0 fallback | ||||||||
| Product page | 96.97% 3 critical · 3 fallback | ||||||||
| Search results | 97.46% 3 critical · 3 fallback | ||||||||
| Account settings | 99.89% 2 critical · 0 fallback |
What people believe, and what is true
Graceful degradation means wrapping calls in try/catch.
The catch block is the last 5% of the work. The other 95% is deciding, per feature, what the user should see — and that is a decision the code cannot make for you.
An optional dependency cannot hurt the request.
It can spend the entire latency budget and exhaust the shared connection pool while being perfectly optional. Optional means "we can proceed without the answer", which only pays off if you also stop waiting for it early.
If the system is degrading, we will see it on the dashboard.
Degradation converts errors into successes. Every standard SLI improves. Only a dedicated degraded-path counter shows it.
Fail-open is the safe default because it keeps the site up.
It keeps the site up by disabling whatever the check was protecting. For a fraud or authorisation check that is a security decision, not an availability one.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Decide in advance which dependencies each feature can survive losing, and what the user sees when it does. Recommendations down should never mean checkout down.
Practical
Build the (feature × dependency) matrix, encode each verdict as data at the call site, give optional dependencies a small dedicated deadline, mark degraded responses, and alert on degradation ratio. Test the fallback paths — they are the least-executed code you own.
Advanced
Degradation is how you escape the multiplication of dependency availabilities. Eight sequential dependencies at 99.9% each give a feature 99.2%; making six of them optional puts the feature back at the availability of the two critical ones. That arithmetic is the actual argument for degradation, and it collapses the moment the failures are correlated through shared infrastructure.
Apply it
- 🔧 Pick one feature in your system and fill in its row of the degradation matrix. For every cell you cannot fill in under a minute, name the person who has to decide it.
- 🔧 Find an optional dependency whose timeout equals the request timeout, and give it its own budget instead.
- ⚡ A cache that backs the "best sellers" fallback is populated by a nightly job that reads from the recommendation service. The recommendation service has been down for 26 hours. What does the fallback serve, and what should it serve?
- 💬 Your product page calls six services. Which of them should be able to take the page down, and how did you decide?
- 💬 The fraud service is unavailable during checkout. Do you accept the order or reject it? What does the answer depend on?
- 💬 Availability is green, error rate is flat, and users say the site "feels broken". What signal are you missing?