Calling Something You Do Not Control
Eight questions every outbound call has to answer, and the fact that a dependency's availability becomes yours the moment you await it.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What do I have to decide before my code calls a system I do not operate?
Checkout must charge a card, validate an address, check a fraud score and send a receipt. Four systems, none of them ours, all of them occasionally unwell.
Import the vendor SDK, call the method, await the result. The SDK handles the details — that is what we are paying for.
The vendor has a bad ten minutes. Every checkout request blocks on their call, workers fill up, and requests that never touch the vendor start failing too (Connection Pool Exhaustion).
- The vendor has a bad ten minutes. Every checkout request blocks on their call, workers fill up, and requests that never touch the vendor start failing too (Connection Pool Exhaustion).
- The SDK's default timeout is longer than your request budget, or absent, so a hung TCP connection holds a worker until something else gives up first (Timeouts).
- The SDK retries internally, your service retries around it, and one user action becomes nine charge attempts (Retries).
- You are rate limited during a traffic spike and every 429 is treated as a generic failure, so the response is to retry harder (Rate Limiting).
- The vendor changes a field. Your parser throws in production on a payload shape their sandbox never returned.
- Nobody can answer "is it them or us?" during an incident, because there is no metric that separates the vendor's latency from your own (Why Is My API Slow?).
What is actually happening
- An outbound call converts an in-process function call into a network conversation with an independent failure domain. It can be slow, it can return garbage, it can succeed and never tell you, and it can do all three within one minute.
- Your availability becomes a function of theirs for every code path that waits on them. Four dependencies each at 99.9% do not give you 99.9% if all four are in the critical path (Failure Propagation).
- The dangerous state is slow, not down. A dependency returning errors quickly is survivable; a dependency taking a very long time to answer converts your bounded worker pool into a queue and takes the whole service with it (Cascading Failure).
- Every call has an ambiguous outcome class: timeouts and connection resets tell you nothing about whether the other side acted. That ambiguity, not the failure itself, is what makes correctness hard (Retries).
- Vendor SDKs embed policy — timeouts, retries, connection pooling, backoff — that you inherit without choosing and usually without reading. It is your policy now regardless of who wrote it.
Eight questions, one table, no blank cells
This table is the deliverable of an integration review. It fits on one page per dependency and it is the difference between a team that knows what happens when the fraud service is slow and a team that finds out.
The column that changes the most answers is the first: whether the request can succeed without this call. A degradable dependency gets a short timeout, no retries in the request path and a defined fallback. A critical one gets a longer budget and an idempotency key, because you cannot simply give up.
| Question | What a good answer looks like | What a blank cell causes |
|---|---|---|
| Critical or degradable? | "Checkout fails without payment. Checkout succeeds without the fraud score, defaulting to manual review." | Everything treated as critical; one vendor blip fails every request. |
| Timeout | Derived from their observed successful-call distribution and our remaining request budget (Timeouts). | A hung call holds a worker indefinitely. |
| Retry | Which error classes, how many attempts, under what budget, inside what deadline (Retries). | Either no resilience, or retry storms during an outage (Retry Storms). |
| Idempotent? | A client-generated key per logical operation, so a retry cannot double-charge (Idempotency Keys). | Retries become duplicate side effects. Money moves twice. |
| Rate limit | Their documented limits, our headroom, and what we do on 429 (Rate Limiting). | 429s treated as errors and retried, guaranteeing more 429s. |
| Auth and rotation | Where the credential lives, who rotates it, how the overlap works (Secrets Are Not Configuration). | An outage at expiry, at an hour nobody chose. |
| Fallback | Cached value, degraded feature, queued for later, or an explicit user-visible error. | The fallback is whatever the exception handler happens to do. |
| Breaker and isolation | Fail fast after sustained failure; a bounded concurrency budget per dependency (Circuit Breakers, Bulkheads). | One slow dependency consumes every worker in the service. |
| Observability | Rate, errors by class, latency histogram, timeouts, in-flight, spans (The Metrics a Backend Must Emit). | "Is it them or us?" takes an hour of an incident to answer. |
The dependency that is slow is worse than the one that is down
This is the least intuitive fact about integrations and the one that causes the largest outages. A dependency returning connection-refused immediately is a clean failure: your code takes the error path, releases the worker, and the rest of the service carries on. A dependency taking a very long time to answer holds a worker for every concurrent request until you have none left.
At that point the symptom is not "payments are failing". It is "the entire API is timing out", including endpoints that never call payments — because the shared resource that ran out was the worker pool, not the vendor (Cascading Failure).
Where the policy lives
Resilience policy has to live somewhere, and the three plausible places have genuinely different trade-offs. The wrong outcome is the fourth one: policy in all three at once, applied multiplicatively, discovered during an incident.
Whatever you choose, the rule that matters is that retries happen at exactly one layer. Three layers each retrying three times is twenty-seven attempts for one user action (Retries).
Where should the resilience policy for this dependency be implemented?
when The default. You need per-operation policy, business-aware fallbacks and error translation.
cost Written once per language and per service; easy to drift between services.
when Simple, well-documented, and its defaults genuinely match your budget.
cost Opaque and often unconfigurable. Its internal retries can be invisible in your metrics and dangerous on non-idempotent calls.
when Many services, many languages, and you want uniform policy without touching each codebase.
cost Policy is far from the code that needs it; it cannot know whether an operation is idempotent, so it cannot retry safely on your behalf.
when Centralised egress control, allowlisting and credential injection.
cost Another hop and another failure domain; per-operation nuance is hard to express (API Gateway in Architecture).
How to build it
Most important first.
- For each dependency, write down eight answers before the first call ships: timeout, retry policy, idempotency, rate limit, auth and rotation, fallback, breaker, observability. The list is the deliverable; a dependency with a blank cell is an incident with a schedule.
- Classify the dependency first: is it critical (the request cannot succeed without it) or degradable (the request can succeed with less)? Almost everything teams call critical is degradable, and the classification decides every other answer.
- Wrap every dependency behind a thin adapter you own. It is where the timeout, the breaker, the metric and the error translation live, and it is what makes the vendor replaceable (The Repository Layer is the same idea for databases).
- Read the SDK's defaults and override them explicitly. An explicit value you chose badly is fixable; an inherited default nobody knows about is not.
- Translate vendor errors into your own taxonomy at the boundary, so application code branches on "retryable", "invalid", "over quota" rather than on a vendor's status string (An Error Taxonomy That Maps Cause to Response).
- Isolate the resources each dependency can consume so one of them cannot starve the others (Bulkheads).
- Test the failure paths, not just the happy one: a fake that can be made slow, erroring and rate-limiting is worth more than a fake that always returns 200 (A Test Strategy Chosen by What Each Layer Can Prove).
What can go wrong
- The mitigation itself failing: a breaker that opens on a blip and stays open, a fallback that returns stale data nobody realises is stale, a retry that doubles a charge.
- A sandbox that behaves nothing like production — faster, more forgiving, differently shaped — so every failure path is untested until it is live.
- An expired credential discovered at 2am because rotation was manual and undocumented (Secrets Are Not Configuration).
- The vendor deprecating a version on their schedule. Their migration deadline is now your sprint plan.
- A dependency that is fast in your region and slow in another, so the failure appears only for some users.
- A "non-critical" call in the request path that was never actually made non-blocking, so its outage is indistinguishable from a critical one.
- A response arriving after your timeout fired: the vendor did the work, you recorded a failure, and a retry is now a second execution (Idempotency Keys).
- Concurrent requests all discovering the dependency is slow and all retrying together, synchronising into a burst (Backoff and Jitter).
- Credential rotation mid-flight, where in-flight calls carry the old secret and new ones carry the new, so both must be valid during the overlap.
- Every response is untrusted input. Validate and bound it — a vendor's bug is your deserialization bug, and an oversized response can exhaust memory as easily as an attacker's can (The Trust Boundary).
- Outbound calls to a URL derived from user input are server-side request forgery, and the vendor integration is where that URL usually comes from (SSRF — When the Backend Fetches a URL).
- Credentials are per-environment, rotatable and never in the repository. A shared production key in a staging config is one careless test away from real money moving (Secrets Are Not Configuration).
- Request and response bodies routinely carry card fragments, tokens and personal data. Log the shape and the identifiers, never the body (Secrets in Logs).
- Pin what you can verify: TLS to a known host, a webhook signature on the way back, an allowlist of egress destinations (Webhook Signature Verification).
- "The SDK handles reliability." The SDK handles serialization and auth. Its timeout, retry and pooling defaults were chosen for a general case that is not yours, and its retries can be actively dangerous on non-idempotent operations.
- "It is a well-known vendor, so it is reliable." Reliability is about what your service does during their bad ten minutes, not about how many bad ten minutes they have.
- "We only call it occasionally, so it does not need a timeout." Frequency has nothing to do with it. One hung call holds one worker forever, and forever is a long time.
- "If it fails we will just retry." Retry is one of the eight answers and it is meaningless without the idempotency answer beside it (Retries).
- "Moving it to a background job removes the dependency." It moves the failure to a consumer where it is quieter, which is often right and is not the same as removing it (Background Jobs).
Operating it
- Per dependency and per operation: request rate, error rate split by class, and a latency histogram of the client-observed duration including connection acquisition (The Metrics a Backend Must Emit).
- A timeout counter that is separate from the error counter. Timeouts and 500s have different causes and different responses, and merging them hides the more dangerous one.
- Spans around every outbound call so a slow request decomposes into "ours" and "theirs" without argument (Tracing From the Backend's Side).
- Saturation of whatever bounds concurrency to that dependency — in-flight calls against the limit. This is the metric that predicts an outage rather than reporting it (Unbounded Concurrency).
- The vendor's own status page and rate-limit headers, recorded as metrics. Their remaining-quota header is free capacity telemetry.
- At 10x, per-request connection setup starts to dominate and connection reuse becomes load-bearing (Keep-Alive and Connection Reuse).
- At 10x, you meet the vendor's rate limits, which converts a capacity problem into a queueing and prioritisation problem (Rate Limit Algorithms).
- At 100x, batch and bulk endpoints stop being an optimisation: N individual calls is a cost and a limit problem simultaneously.
- Cost scales with call volume in a way in-process work does not. A per-call price makes caching and coalescing an economic decision as much as a performance one (Request Coalescing).
- An adapter layer per dependency is real code to write and maintain, and for a single call to a stable vendor it is ceremony. The threshold is roughly when the second caller or the second failure mode appears.
- Fallbacks add a second behaviour to reason about and test. A stale-but-serving fallback can be worse than an error if the staleness is invisible to the caller.
- Strict timeouts abandon work that would have succeeded, and every one of those is a user-visible failure you chose.
- Isolating resources per dependency means unused capacity in one pool while another queues. That waste is the price of containment (Bulkheads).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe eight questions apply to any outbound dependency in any language: payment APIs, internal services, object storage, an SMTP relay, even your own database.
- RUNTIME-SPECIFICWhat "one hung call" costs differs by runtime. On a thread-per-request model it blocks a thread and the pool has a hard size; on a single-threaded event loop the await itself is cheap but unbounded in-flight promises grow memory and every timer competes for the same loop (Backend Runtime Models).
- FRAMEWORK-SPECIFICSome HTTP clients apply a total-request deadline; others apply separate connect, read and idle timeouts that can sum to far more than you intended. Read the specific client's semantics rather than assuming the number you set is the ceiling.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — independent failure domains and why availability composes multiplicatively across a synchronous call chain.