A slow recommendations widget took down checkout

• recommendations-svc p99 went from 180 ms to 45 s (its vector store was compacting).
• Within three minutes *every* storefront endpoint returned 503 — including checkout, cart and search, none of which call recommendations.
• Storefront CPU was 12%; memory flat; the server thread pool was at 200/200.
• The load balancer marked all six storefront instances unhealthy at 15:21.

# storefront thread dump, 15:21:10   (server pool: max 200 threads)
  187 threads   WAITING   socketRead0  ← recommendations-svc:8080  GET /recs?user=…   (oldest: 41 s)
    9 threads   WAITING   socketRead0  ← order-svc:8080
    4 threads   RUNNABLE  (accept queue: 1,140 pending connections; /health queued behind them)

# outbound HTTP client (shared by every dependency)
http.client.connectTimeout        = 2000
http.client.readTimeout           = 0        # 0 = wait forever
http.client.maxConnections        = 200      # one pool for all routes
http.client.maxConnectionsPerRoute = 200     # a single route may take the entire pool

# 1-minute metrics
time    recs p99     recs rps   storefront 503   busy threads
15:18   180 ms       62         0 %              14 / 200
15:19   6,100 ms     61         3 %              88 / 200
15:20   45,000 ms    60         38 %             196 / 200
15:21   timeout      —          100 %            200 / 200   (LB: 0/6 healthy)