APIs & GatewaysdesignExpert

Every deploy touches the gateway

Symptoms

  • Gateway p99 has gone from 90 ms to 210 ms over six months; adding instances (6 → 12) changed nothing.
  • 70% of pull requests across five product teams modify the gateway repository; it is 48,000 lines and deploys about three times a day.
  • A bug in a loyalty-tier calculation inside the gateway returned 500 for *every* route — including ones that have nothing to do with loyalty — for 12 minutes.
  • Seven of the last nine incidents involved a gateway deploy.
client ──► gateway ──────────────────────────────────────────────────────────► services
             │  tls, auth (2 ms), rate limit (1 ms), routing            ← edge concerns
             │
             │  handler /checkout/summary  (one of 61 handlers with cross-service logic)
             │     GET user-svc/users/:id                        38 ms  ┐
             │     GET cart-svc/carts/:id                        41 ms  │ sequential
             │     POST pricing-svc/quote  (300-line body builder) 52 ms │ (sum: 160 ms)
             │     GET loyalty-svc/tier                           29 ms  ┘
             │     if (tier === 'gold' && cart.total > 100) discount = 0.05 * cart.total   ← business rule
             │     merge → JSON shaped for the iOS app (owned by the mobile team)
             ▼
gateway repo        48,200 LOC · 61 route handlers · CODEOWNERS: (empty) · unit tests: middleware only
deploys / 30 d      gateway 84 · checkout-svc 9 · pricing-svc 6 · user-svc 4
incidents / 90 d    9 total · 7 involved a gateway deploy · 1 took all routes down (loyalty NPE)

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Gateway CPU and instance count
Auth and rate-limit middleware
The /checkout/summary handler
Ownership of the gateway repository
Upstream service latencies
Recent framework upgrade
Network and DNS