External Calls Inside a Transaction
A five-second payment call between BEGIN and COMMIT holds a pooled connection and every lock the transaction took, for five seconds, on every request.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What does it actually cost to call another system while a database transaction is open?
Checkout writes an order and charges a card. The charge is an HTTPS call to a payment provider, and it is naturally written in the middle of the handler where the order is created.
Open the transaction, insert the order, call the payment API, and commit if the charge succeeded. That way a failed charge rolls back the order automatically — which is exactly the behaviour we want.
The provider takes five seconds. For those five seconds this request holds one pooled connection, and no other request can use it. With a pool of 20, the service can complete at most 20/5 = four checkouts per second, no matter how many CPUs or instances you have (Connection Pools).
- The provider takes five seconds. For those five seconds this request holds one pooled connection, and no other request can use it. With a pool of 20, the service can complete at most 20/5 = four checkouts per second, no matter how many CPUs or instances you have (Connection Pools).
- It also holds every row lock the earlier writes took. Any other transaction touching those rows — the stock row, the customer row — waits for the payment provider to answer (Low CPU, High Latency: Lock Contention).
- On an MVCC engine it holds a snapshot open, so dead row versions across the whole database cannot be cleaned up while it waits (MVCC: Multi-Version Concurrency Control).
- If the call has no timeout, none of those durations are bounded. A provider that hangs holds your connection until something else gives up (Timeouts).
- When the provider is slow, the failure is service-wide: endpoints with nothing to do with checkout start timing out, because the pool is full of requests waiting on HTTP.
- The rollback does not even do what it promised. If the charge succeeded and the commit then failed, the customer is charged for an order that does not exist — and
ROLLBACKcannot un-charge a card.
What is actually happening
- A transaction holds resources for its wall-clock duration, not for the time it spends doing database work. The database cannot tell the difference between "computing" and "waiting on someone else's API".
- The resources held are specific and countable: one connection from the pool, every lock acquired so far, and on MVCC engines a snapshot that pins the cleanup horizon.
- Throughput follows directly. A pool of N connections and a mean hold time of T seconds gives a ceiling of N/T transactions per second. Putting a network call inside the bracket increases T by the remote system's latency — the one variable you do not control.
- Worse, T becomes a function of someone else's availability. Your capacity is now coupled to their p99, and their bad day is your outage (Failure Propagation).
- The queueing is silent. Requests waiting for a connection are not errors and not slow queries; they are simply waiting, and only pool metrics show it (Connection Pool Saturation: Waiting in Front of an Idle Database).
- The same argument applies to anything slow inside the bracket: a cache miss to a remote cache, an S3 upload, a large in-process computation, a
sleep, or waiting on user input.
What is held, and for how long
Make the cost concrete before arguing about design. Between BEGIN and COMMIT, this request owns three things: one of the pool's connections, every row lock its writes have taken, and a snapshot that prevents cleanup of old row versions. All three are held for wall-clock time, and the payment provider decides how much of that there is.
The arithmetic is the argument. A pool of 20 and a five-second call is a hard ceiling of four checkouts per second. Adding application instances does not help, because each new pool adds connections to a database that has its own limit. The only lever that works is making the bracket shorter.
Before, or after — there is no inside
Once the call is out of the bracket you must choose which failure you would rather handle, and the choice is real. Call before the transaction and you can fail cleanly, but you may have caused an effect for an order that is never created. Call after the commit and the order is definitely real, but the call may never happen unless something durable is responsible for it.
For payments the standard resolution is to split the provider interaction: authorize before, commit, capture after. Authorization is reversible and cheap to abandon; the capture is driven by a durable job that can retry.
await withTransaction(async (tx) => {
const order = await createOrder(tx, input)
await decrementStock(tx, input.items) // locks taken here
const charge = await payments.charge(card, order.total) // 5 s, locks held
await recordPayment(tx, order.id, charge.id)
})
// pool slot + row locks held for the provider's latency, per request// 1. before BEGIN: no locks, no pool slot beyond the HTTP client
const auth = await payments.authorize(card, quote.total, {
idempotencyKey: input.requestId,
})
// 2. short bracket: database writes only
const order = await withTransaction(async (tx) => {
const o = await createOrder(tx, input, auth.id)
await decrementStock(tx, input.items)
await tx.query('INSERT INTO outbox (topic, payload) VALUES ($1, $2)',
['payment.capture', JSON.stringify({ orderId: o.id, authId: auth.id })])
return o
})
// 3. after commit: a worker captures, retries, and is idempotentThe bracket in the second version contains only database work, so its duration is a function of your own queries and nothing else. The provider's latency no longer sets your throughput ceiling. The idempotency key makes the authorization safe to retry, and the outbox row makes the capture durable without extending the transaction (Idempotency Keys).
What it looks like when it happens to you
This bug is usually reported as "the database is slow" or "the site is down", because the symptom is service-wide and appears nowhere near the code that causes it. The rows below map each trigger to what you actually see on a dashboard.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Payment provider p99 rises | Every endpoint slows, then 5xx; database CPU near idle | Pool slots held by requests waiting on HTTP | Move the call out of the bracket; bulkhead the client (Bulkheads) |
| External call with no timeout | Connections stuck "idle in transaction" indefinitely | Nothing bounds the wait | Aggressive client timeout plus a database idle-in-transaction timeout as a backstop |
| Locks held across the call | Unrelated writes to the same rows time out | Row locks taken before the call are held until commit | Take locks as late as possible; keep the call outside entirely |
| Charge succeeds, commit fails | Customer charged, no order exists | Two systems, one of which cannot roll back | Authorize-then-capture with an idempotency key; reconcile daily (The Dual Write Problem) |
| Transaction retried after a conflict | Duplicate charges | The retry re-ran the external call | Never place non-idempotent effects inside a retriable bracket |
| Provider fully down | Total service outage, not just checkout | Shared pool consumed by one endpoint | Circuit breaker plus a per-dependency concurrency limit (Circuit Breakers) |
How to build it
Most important first.
- Do the external call before
BEGINor afterCOMMIT. Those are the only two safe places, and choosing between them is a design decision about which failure you prefer. - Prefer authorize-then-commit-then-capture for payments: authorize before the transaction, record the authorization id inside it, capture afterwards. The authorization is reversible; the order write is atomic.
- When the call must follow the write, put an outbox row or a durable job inside the transaction and make the call from a worker afterwards (The Transactional Outbox).
- Give every external call an aggressive timeout, and treat that timeout as the upper bound of your transaction duration if you cannot get the call out of the bracket at all (Timeouts).
- Bound the concurrency of the external call separately from request concurrency, so a slow dependency cannot consume the whole pool (Bulkheads).
- Make anything that runs after the commit idempotent, because it will be retried (Idempotency in Backends).
- Assert it in review and, where you can, in code: a check that no HTTP client is invoked while a transaction is open catches this permanently.
What can go wrong
- Pool exhaustion driven entirely by a third party. The database is idle, your CPU is idle, and every request is waiting (Connection Pool Exhaustion).
- Lock waits cascading into deadlocks, because transactions that would have finished quickly are now overlapping (Deadlocks in Application Code).
- The charge succeeds, the commit fails: money taken, no order. This is the failure the design was supposed to prevent, and it is still possible.
- A retry of the whole transaction re-issuing the external call, charging twice (Duplicate Detection).
- A provider outage becoming a total outage of your service, rather than a failure of one endpoint (Cascading Failure).
- The mitigation failing too: an aggressive timeout returns an error while the remote side still processes the request, so you have an unknown outcome and a rolled-back transaction.
- The timeout race: you give up, the provider completes. The charge exists and your transaction rolled back — an unknown outcome that only reconciliation can resolve (At-Least-Once Delivery).
- Overlapping transactions caused by the extra duration make previously-impossible lock cycles possible, so deadlock rates rise as the dependency slows (Deadlocks in Application Code).
- A retried transaction re-invoking the external call concurrently with the original attempt, if the original never actually failed (Idempotency Keys).
- Holding transactions open on caller-controlled timing is a denial-of-service lever: an attacker who can make the external call slow — or who controls a webhook target you call synchronously — can drain the pool (Resource Limits).
- Never call a caller-supplied URL from inside a transaction. It is both an SSRF and a lock-holding primitive in one (SSRF — When the Backend Fetches a URL).
- Payment credentials and provider responses must not be written to logs inside the error path that dumps transaction context (Secrets in Logs).
- "The transaction gives us atomicity across the database and the payment provider." It does not. No transaction spans two systems here; you have a database transaction with a network call inside it.
- "It is only slow when the provider is slow." That is exactly the problem: your capacity is defined by someone else's latency.
- "The connection is idle while we wait, so it is free." It is checked out of the pool. Idle-but-held is the worst state a connection can be in.
- "We added a timeout, so it is fine now." A timeout bounds the damage; it does not remove it. A 3-second timeout with a pool of 20 still caps you at roughly 7 transactions per second in the bad case.
- "Read-only transactions are fine." A read-only transaction still holds a connection and, on Postgres, still pins the snapshot horizon.
Operating it
- Transaction duration and external-call duration on the same trace. If the HTTP span sits visibly inside the transaction span, the diagnosis is complete (Tracing From the Backend's Side).
- Pool waiting count and acquire wait time. A rise correlated with a dependency's latency is the signature of this bug (Connection Pool Saturation: Waiting in Front of an Idle Database).
- Longest-running transaction on the database, alerted. It catches this and every other held-bracket problem.
- Sessions idle in transaction: a connection that is inside a transaction and running no query is, by definition, waiting for your application to do something else.
- The arithmetic gets worse linearly with the dependency's latency and stays fixed as you add application instances — unless you also add connections, which the database will not allow indefinitely.
- At 10x traffic, an in-transaction external call is usually the first thing that falls over, because it consumes the scarcest resource with the least local benefit.
- Removing the call from the bracket often multiplies throughput by more than any query optimisation available, because it reduces T rather than N.
- At larger scale this pattern is prohibited outright, usually by a lint rule or a wrapper that refuses to make HTTP requests while a transaction is open.
- Moving the call outside the transaction means you can no longer roll back the write because the call failed. You gain resource safety and take on explicit failure handling.
- Authorize-before / capture-after adds a two-phase interaction with the provider and an authorization that can expire.
- An outbox plus a worker adds machinery and an eventual-consistency window, and it is the only shape that is both resource-safe and reliable (The Transactional Outbox).
- Short timeouts convert waiting into unknown outcomes, which must then be reconciled. Uncertainty is cheaper than a locked pool, but it is not free.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALTrue of every pooled database client. The resource being held is the pool slot, and every runtime and engine has one.
- DATABASE-SPECIFICPostgres additionally holds back the xmin horizon for the transaction's life, so a single long transaction degrades vacuum across the whole database, not just the touched tables. InnoDB instead grows the undo history list, lengthening the version chains consistent reads must traverse. Both are real; the diagnostic you reach for differs.
- RUNTIME-SPECIFICOn an event-loop runtime the awaiting request does not occupy a thread, so the process happily accepts thousands more requests while holding the pool empty — the queue grows silently. A thread-per-request runtime stalls a thread too, which makes the saturation more visible sooner but caps throughput just as hard.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.