API latency went from 100 ms to 4 seconds

• orders-svc is a thread-per-request service: 8 replicas, 200 worker threads each, one Postgres pool of 20 connections per replica.
• Release 2024.31 added a "delivery ETA" line to the order detail page. The ETA comes from a carrier API (carrier-api.example) that the team had not integrated before.
• The carrier’s status page has shown "degraded performance" since 13:50 — thirty minutes before the deploy.
• Available: RED metrics per endpoint, container CPU/memory, Postgres dashboards, and OpenTelemetry traces sampled at 10%.
• Rolling back is possible but the on-call wants to know *why* before deciding, because the ETA feature is contractually due this week.

# trace 4d1f…  GET /orders/88213   total 4 112 ms   status 200
span                                            start      dur
orders-svc  GET /orders/:id                     0 ms       4 112 ms
  auth.verify                                   1 ms       3 ms
  pg  SELECT … FROM orders WHERE id = $1        5 ms       4 ms
  pg  SELECT … FROM order_items WHERE …         10 ms      6 ms
  eta.lookup                    (new in 2024.31) 17 ms     3 862 ms
    http POST carrier-api.example/v2/eta        18 ms      3 860 ms   200
  render                                        3 881 ms   2 ms

# orders-svc, one replica, 14:10 → 14:30
                            14:10   14:15   14:20   14:25   14:30
p50 latency (ms)              108     112     980   3 900   4 100
worker threads busy (/200)     14      15     140     200     200
requests queued                 0       0      12     310     640
CPU (cores)                   0.6     0.6     0.5     0.4     0.4
pg pool in use (/20)            3       3       3       4       3
outbound conns to carrier       0       0     138     200     200