Treating External Systems as What They Are
An external system can fail, be slow, change, rate-limit you, and repeat itself. None of those is a bug in it; they are properties of being outside. The interface you design around it either accounts for all five or discovers them one incident at a time.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Everything outside the boundary can fail, stall, change, throttle you and send things twice. What does each of those do to the interface you design around it?
The payment integration works in test mode and I am about to call it done. A colleague asked what happens if the provider is down for an hour, and then what happens if they are up but slow, and then what happens if they send the confirmation twice. I had one answer for the three questions, and it was "it probably won't".
Treat the provider like a function. Call it, get a result, carry on — because in test mode that is exactly what happens, and the provider's documentation is written in the voice of something that always works. Failure handling can be added when a failure is seen.
The first failure is seen in production, by a customer, and the "handling" is a stack trace. The reflex postponed the design of the failure paths to the moment with the least time to design them.
- The first failure is seen in production, by a customer, and the "handling" is a stack trace. The reflex postponed the design of the failure paths to the moment with the least time to design them.
- The five properties are treated as one property, "unreliable", and given one response: a retry. A retry is right for a transient failure, wrong for a timeout on a charge that may have succeeded, and actively harmful under rate-limiting.
- "Change" is not on the list at all, because nothing in test mode changed. The provider's deprecation email goes to a mailbox nobody reads, and the field disappears on schedule.
- The integration has been declared finished at exactly the point where the interesting part of the interface — what to do when the other side does not behave — has not been started.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Give each property of an external system its own question, because each one has a different answer. It can fail: what does the caller do when it says no? It can be slow: how long does the caller wait, and what does it say when the wait runs out? It can change: where does their vocabulary live in your code, and how would you find out before they change it? It can rate-limit: what happens to the calls you are not allowed to make? It can repeat: is every message it sends you safe to receive twice?
- Answer each question at the interface, not in the caller. The adapter gets the timeout, the idempotency key and the translation; the caller gets three outcomes and the state that a deferred resolution needs. A caller that handles the provider's failure modes directly has moved the boundary into itself.
- Distinguish "it said no" from "it said nothing". A decline is an answer and the interface passes it through. A timeout is the absence of an answer, and the only honest thing the interface can do is report UNKNOWN and arrange to find out later. Retrying a charge after a timeout, without an idempotency key, is how customers get charged twice.
- Decide what the system does while the external one is unavailable: refuse the operation, queue it, or degrade to something without it. Each is a product decision as much as a technical one, and each changes the state model.
Five properties, five different answers
Each row is one thing an external system can do to you, written for the payment provider on checkout's response path. The response column is different in every row, which is the point: a system that answers all five with "retry" has answered one of them.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| It fails (declines, errors) | Customer cannot pay | Card declined, provider error | Pass DECLINED through; order stays unpaid; customer can try again. Do not retry a decline. |
| It is slow (no answer in time) | Customer waits; money may or may not have moved | Provider latency, network | Bounded wait, then UNKNOWN; order becomes PENDING_CONFIRMATION; reconcile later by order id. Never retry without the idempotency key. |
| It changes (fields, semantics) | Adapter breaks, or worse, silently maps wrongly | Provider deprecation, version bump | Vocabulary confined to the adapter; contract test in CI or on a schedule; a named owner on their changelog. |
| It throttles you | Some checkouts refused | Rate limit exceeded, often by your own retries | Refuse honestly on the response path; back off; never retry into a limit. |
| It repeats itself | Same outcome delivered twice | Webhook redelivery | Record event ids; the second arrival is a no-op. Ordering is not guaranteed either — a later event may arrive first. |
One call, from the adapter's side
The pipeline is what one outbound call to an external system goes through once the five questions are answered. Every step has a way of failing, and the failure is named so the caller gets an outcome rather than an exception. Read it as the adapter's job description.
- 1Translate
Build the provider's request from your nouns; attach the idempotency key derived from the order.
fails by Key derived from something that changes on retry, so the retry is a new request.
- 2Bound the wait
Send with a timeout chosen from what the user can bear.
fails by Timeout copied from the provider's example; the customer has gone by the time it fires.
- 3Classify the result
Answer → PAID or DECLINED. No answer → UNKNOWN. Throttled → REFUSED, with no automatic retry on the response path.
fails by A timeout reported as DECLINED; the customer retries and is charged twice.
- 4Retry only what is safe
Transient failure on an idempotent request: retry with backoff, within a budget.
fails by Retrying into a rate limit; retrying a decline.
- 5Record and hand back
Store the provider's reference for reconciliation; return the outcome in your vocabulary.
fails by Storing their status string as your state, so their change becomes your bug.
Integrating the provider, as testable pieces
The reflex calls the integration done when a test-mode charge succeeds. The decomposition says what "done" contains once the five properties are taken seriously; each leaf is something you can observe, and most of them are things the happy path never exercises.
- ├Happy path
- └Test-mode charge succeedstestable A charge for a test order returns PAID and the order shows paid.
- └Decline is passed throughtestable A test card that declines produces DECLINED and an unpaid order with the reason recorded.
- ├Slow— the property test mode never shows
- └Timeout produces UNKNOWNtestable With the adapter's timeout forced short, checkout writes PENDING_CONFIRMATION and shows the honest sentence.
- └Reconciliation resolves ittestable A job later asks the provider by order id and moves the order to paid or unpaid.
- ├Repeat
- └Duplicate confirmation is a no-optestable Delivering the same webhook event twice leaves one payment record and one state change.
- └Out-of-order events do no harmtestable A "succeeded" arriving after a "refunded" for the same charge does not move the order backwards.
- ├Change and throttle
- └Contract testtestable A scheduled test against the provider's test mode fails when their response shape changes.
- └Rate limit handledtestable A simulated throttle response yields REFUSED, no automatic retry, and a customer-visible message.
The happy-path leaves are the ones the reflex built. The others are the integration.
How to do it
Most important first.
- For each external system, write the five questions and answer them in the adapter: fail, slow, change, throttle, repeat (External Systems Fail).
- Set a timeout on every external call, chosen from what the user can bear, not from what the provider suggests. Give timeouts their own outcome (Timeouts).
- Retry only what is safe to retry: idempotent requests, with an idempotency key, with backoff, with a budget. A retry against a rate limit is a request to be limited harder (Retry Storms: The Load You Generated Yourself).
- Make every inbound message safe to receive twice, by recording what you have seen (Duplicate Requests).
- Find where their vocabulary lives in your code and confine it. Subscribe to their changelog with a person's name on the subscription.
- Decide the unavailable-mode behaviour explicitly: refuse, queue or degrade — and make the state visible to the user.
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Payment provider, five answers. Fail: DECLINED is passed to checkout, which tells the customer and keeps the order unpaid. Slow: the adapter waits a bounded time, then returns UNKNOWN; checkout writes the order as PENDING_CONFIRMATION and shows an honest sentence; a reconciliation job asks the provider later using the order id. Change: their types appear only in the adapter; a contract test against their test mode runs in CI so a change fails a build instead of a customer. Throttle: checkout is on the response path, so it cannot queue; it refuses with "try again shortly" and the retry is the customer's, not the code's. Repeat: the confirmation handler records each event id and ignores a second arrival.
- Email sender, five answers, and they are different. Fail: retry later; the order stands. Slow: nobody is waiting; the worker's timeout is generous. Change: the adapter. Throttle: the worker slows down; email is allowed to be late. Repeat: a receipt sent twice is annoying and harmless, so no deduplication. Same five questions, five cheaper answers, because the receipt is after the response and nothing depends on it.
- The AI assistant's model API. Fail: show "I could not answer that right now" and log. Slow: stream what arrives and cut off at a bound. Change: the model can change its behaviour without changing its API — a property not on the original list, discovered by asking "change of what?" — so evaluation runs on a fixed question set are the contract test. Throttle: queue questions with a visible position. Repeat: not applicable; we call it, it does not call us.
How you know it worked
What now exists that did not before, and what question you can now ask.
- For each external system, there are five written answers, and they differ across systems in ways you can explain.
- Every external call has a timeout, and the timeout produces an outcome the caller handles on purpose.
- A message from outside arriving twice changes nothing, and there is a test that proves it.
- The system has a named behaviour for "the provider is unavailable", and the user can tell that state from an error.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?For this external system: what do I do when it says no, when it says nothing, when it changes, when it throttles me, and when it repeats itself?
- ?Is this request safe to retry — and if the answer depends on an idempotency key, do I send one?
- ?What does the system do while this external system is unavailable: refuse, queue, or degrade?
- ?How would I find out about a change on their side before a customer does?
What can go wrong
- Every external system gets the full treatment regardless of what depends on it. The receipt email gets a circuit breaker, a dead-letter queue and a reconciliation job. The five questions are asked everywhere; the answers should cost in proportion to what the failure costs.
- The retry is added and the idempotency key is not. Now the failure mode "slow" produces the failure mode "charged twice", and the fix has made things worse than the reflex.
- Timeouts are set from the provider's documentation rather than from the customer's patience, so the checkout waits longer than anyone will and then reports UNKNOWN to an empty browser.
- "Change" is answered with an adapter and no early-warning. The adapter contains the change; nothing detects it, and the contract breaks on the provider's schedule.
- Five answers per external system is real design work, and for an external system nothing important depends on, most of the answers are "do not care" written slowly.
- A timeout short enough to respect the customer's patience produces more UNKNOWN outcomes, each of which needs a state, a sentence and a reconciliation. Honesty about slowness is paid for in state.
- A contract test against the provider's test mode makes CI depend on an external system, which is exactly the thing the lesson warns about; teams that want a deterministic build run it on a schedule instead.
- "External systems are unreliable, so I should retry everything." Retrying is safe only for idempotent requests, and useful only for transient failures. A retry against a decline, a rate limit or a timeout on a non-idempotent charge is respectively pointless, harmful and dangerous.
- "A circuit breaker solves this." A breaker stops you hammering a system that is down; it does not tell you what to show the customer, and it does not resolve the charge that timed out before the breaker opened. It is one answer to one of the five questions.
- "Internal services do not need this." The moment an internal service is on the far side of a network call it can fail, stall and repeat; what changes is that you can fix it. The five questions apply with softer answers, not no answers.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALFail, slow, change, throttle, repeat are properties of anything across a boundary you do not control, from a payment provider to a public dataset to a model API.
- DOMAIN-SPECIFICThe cost of each answer depends on what the failure costs. Money crossing the boundary makes "slow" and "repeat" the expensive questions; for an analytics dashboard reading an upstream source, "change" is the expensive one and "repeat" barely matters.
- ILLUSTRATIVEThe provider, the email sender, the model API and their five answers are invented to show the shape; a real provider's documentation has specifics that overrule any of them.
Where the depth lives
This domain asks the question and hands the answer off by name.