Intermediate · K8s Runtime
Health checks that take the service down instead of protecting it
A service has one health endpoint used for both liveness and readiness. It checks the database, the cache and two downstream APIs, and returns 503 if any is unavailable. During a brief database slowdown, every instance was restarted, and the service was unavailable for far longer than the database was.
The fix that looks right
Trap
Making the health check more thorough after the incident — adding checks for more dependencies so problems are "caught earlier". It increases the number of external conditions that can kill every instance at once, and every added check is another synchronised failure mode. The instinct is right and the direction is backwards: the fix is fewer things able to kill the process, and more precision about which things should merely stop traffic.