Distributed SystemsdebuggingIntermediate
A third of internal calls fail for five minutes after every scale-in
Symptoms
- Callers of inventory-svc see ~33%
ECONNREFUSEDfor about five minutes, then it clears — several times every evening. - Each burst starts within seconds of the autoscaler removing inventory-svc instances.
- The service registry lists 9 healthy instances; the orchestrator shows 6 running.
- Retries help a little but not much: the retry often goes to another dead address.
# registry: inventory-svc, 21:14:30 # autoscaler log
10.0.4.11:8080 passing last heartbeat 21:14:05 21:09:58 scale-in: terminate 10.0.4.17, .18, .19
10.0.4.12:8080 passing last heartbeat 21:14:10 21:09:58 SIGTERM sent
10.0.4.13:8080 passing last heartbeat 21:14:02 21:10:03 SIGKILL (grace period 5 s)
10.0.4.14:8080 passing last heartbeat 21:14:09
10.0.4.15:8080 passing last heartbeat 21:14:11
10.0.4.16:8080 passing last heartbeat 21:14:07
10.0.4.17:8080 passing last heartbeat 21:09:40 ← terminated 4 min 50 s ago
10.0.4.18:8080 passing last heartbeat 21:09:41 ← terminated
10.0.4.19:8080 passing last heartbeat 21:09:38 ← terminated
# inventory-svc registration
check: { type: ttl, ttl: 300s, deregister_critical_service_after: 600s }
heartbeat: every 60 s from the app · shutdown hook: none · SIGTERM handler: none
# order-svc client: resolve every 30 s → 9 addresses → round robin
21:14:31 → 10.0.4.17 ECONNREFUSED 21:14:31 retry → 10.0.4.18 ECONNREFUSEDInvestigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Autoscaler policy
Heartbeat TTL
Client-side resolution cache
Network policy / firewall
Registry cluster health
Client retry policy
Recent changes