Operating a Load Balancer
The algorithm matters less than the health check, the connection lifetime and the capacity that remains when a backend leaves. Most balancing incidents are about membership, not distribution.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
My traffic is not evenly distributed and removing a backend caused errors. What is the load balancer actually doing?
Traffic has to be spread across a changing set of backends, without sending work to ones that cannot serve it, and without dropping work when the set changes.
Pick round-robin, add the backends, point a health check at /health, and the load balancer will distribute traffic evenly and route around failures.
Round-robin at the connection level is not round-robin at the request level. With connection reuse, one balancing decision can serve thousands of requests, so a small number of long-lived connections pins traffic to a small number of backends (Keep-Alive and Connection Reuse in the networking view).
- Round-robin at the connection level is not round-robin at the request level. With connection reuse, one balancing decision can serve thousands of requests, so a small number of long-lived connections pins traffic to a small number of backends (Keep-Alive and Connection Reuse in the networking view).
- Even distribution of requests is not even distribution of work. If requests differ in cost, an equal share of requests is an unequal share of load (Hot Keys: When Aggregate Metrics Hide a Saturated Node in the observability view).
- A health check pointed at the same endpoint as everything else inherits its problems: too shallow and it keeps failing backends in rotation, too deep and a shared dependency removes every backend at once (Probes: Readiness, Liveness and Startup).
- Removing a backend removes its capacity. The remaining backends absorb its share, and if there was no headroom, removing one unhealthy backend is what pushes the rest over (Cascading Failure: When the Response to Failure Causes More Failure in the backend view).
- A newly added backend starts cold — empty caches, unwarmed runtime, no established connections — and sending it a full share immediately gives it the worst latency of any backend right when it is least able to cope (JIT and Warm-Up: The First Thousand Requests Are a Different Program in the observability view).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A load balancer is three things operating together: a membership list maintained by health checks, a selection rule applied per connection or per request, and a connection model that determines how long a selection lasts.
- L4 balancing selects a backend per connection and then forwards bytes. L7 balancing terminates the connection, understands requests, and can select per request — which is what makes real balancing, retries and header-based routing possible (Load Balancers: L4 vs L7 in the networking view).
- Health checks have thresholds in both directions: consecutive failures to remove, consecutive successes to return. Both matter — the first decides how long a broken backend keeps receiving traffic, the second decides how quickly a flapping backend comes back.
- The load balancer's health check is a separate opinion from the application's readiness. Where both exist they must agree, or a backend is in one list and out of the other and behaviour depends on which layer you ask (Service Discovery in Operation).
- Deregistration has a delay by design: the backend is removed from selection, and existing connections are allowed to finish. Skipping or misconfiguring that delay is what makes deploys drop requests (Draining: Stopping Without Dropping).
- The load balancer is itself a capacity limit — on connections, on new connections per second, on bandwidth — and it is a shared component, so its saturation is everyone's problem at once (Saturation: The Reading Utilization Cannot Give You in the observability view).
L4 and L7 are different operational objects
The layer decides what the balancer can see, and therefore what it can do about a failure. This is not a performance comparison — it is a list of the operations that are simply unavailable at L4.
| Concern | L4 (connection) | L7 (request) | Why it matters operationally |
|---|---|---|---|
| Balancing granularity | Per connection | Per request | Connection reuse makes L4 distribution arbitrarily uneven |
| Retries | Not possible — bytes already forwarded | Can retry an idempotent request on another backend | Decides whether one bad backend produces user-visible errors |
| Health check depth | Usually a TCP connect | An application request with a status code | A TCP connect succeeds on a process that cannot serve anything |
| Routing | By port only | By host, path, header | Everything ingress does needs L7 (Operating the Edge) |
| TLS | Passed through | Terminated here, usually | Decides where the certificate lives (Certificates as an Operational Object) |
| Draining | Wait for connections to close | Wait for in-flight requests, refuse new ones | L7 draining is far more precise (Draining: Stopping Without Dropping) |
| Observability | Connections and bytes | Status codes, latency, per-route detail | Determines whether the balancer can tell you anything during an incident |
How deep should the health check be?
This is the setting that decides both how quickly a bad backend is removed and how completely a shared failure empties the pool. It has no default answer, and the failure modes at the two extremes are very different in kind.
The backends depend on a shared database. What does the check verify?
when L4 balancing, or the application genuinely cannot fail in a way that keeps the socket open.
cost A process that accepts connections and returns errors stays in rotation indefinitely, and every request routed to it fails.
when The common default: proves the process is running and can serve a request.
cost Does not distinguish a backend that is serving from one that is serving errors — but it also cannot empty the pool on a shared failure.
when You need the check to reflect real capability and can accept the pool emptying if the dependency fails.
cost A shared dependency outage removes every backend at once, turning degradation into total unavailability (Probes: Readiness, Liveness and Startup).
when The application can distinguish what it can and cannot serve and report accordingly.
cost The most correct answer and the most application work; it also needs a balancer that can act on more than healthy or unhealthy.
Membership failures, which is most of them
Very few load balancer incidents are about the selection algorithm. They are about who is in the pool, when they entered or left, and how much capacity was left afterwards.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Backend accepts connections but cannot serve | A fraction of requests fail with no failing health check | The check is shallower than the failure | Deepen the check to an application response, deliberately and with the pool-emptying risk in mind |
| Shared dependency fails | Healthy target count drops to zero; total outage | Every backend evaluates the same deep check | This is the health check causing the outage; separate liveness of the pool from health of the dependency |
| Long-lived connections | Traffic heavily skewed; one backend saturated | Balanced per connection, and connections rarely end | Balance per request, or bound connection lifetime (Keep-Alive and Connection Reuse in the backend view) |
| Scale-in or deploy | A burst of errors at every capacity change | Deregistration delay shorter than in-flight request duration | Set the delay above your longest normal request (Draining: Stopping Without Dropping) |
| New backend added | Latency spike attributed to the deploy | A cold instance receiving a full share immediately | Enable slow start where available; warm caches at startup |
| One backend degraded, others healthy | Errors rise but no target is removed | Thresholds require more consecutive failures than the flapping produces | Tune thresholds; a check that never removes anything is not a check |
| Retries enabled at the balancer | Overload gets worse rather than better | Each failing request becomes several against an already saturated pool | Bound retries and add jitter; retry budgets, not unlimited retries (Retry Storms: The Load You Generated Yourself in the backend view) |
| Balancer at its own limit | Connection failures with every backend healthy | Connection or new-connection-rate ceiling reached | Treat the balancer as capacity to plan, not as infinite (Building a Capacity Model) |
How to do it properly
Most important first.
- Fix membership before tuning the algorithm. Skewed traffic is far more often connection reuse or a health-check disagreement than a bad selection rule.
- Balance per request at L7 for anything where distribution matters, or bound connection lifetimes so that L4 selection is re-made regularly.
- Make the load balancer health check and the application readiness definition the same thing, or document precisely why they differ.
- Size for the loss of a backend — and of a zone. If losing one backend saturates the rest, the balancer is not providing availability, only distribution (Capacity During Failover).
- Use slow start or gradual traffic increase for new backends where the platform offers it, so a cold instance is not immediately handed a full share.
- Watch per-backend distribution, not just the total. An aggregate graph looks healthy while one backend takes most of the traffic.
- Treat the load balancer's own limits as capacity you must plan: connection counts, new connections per second, and target counts are real ceilings (Building a Capacity Model).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The load balancer is a shared chokepoint: a health-check misconfiguration can empty the pool for every service behind it, and its own connection limits fail traffic regardless of backend health. What contains an individual backend problem is the health check working correctly — which is the same mechanism that causes the total failure when it is wrong.
What can go wrong
- Health check too shallow: a backend that accepts connections but cannot serve stays in rotation and every request routed to it fails.
- Health check too deep: a shared dependency fails, every backend fails its check, and the entire target group empties — a total outage caused by the health check (Probes: Readiness, Liveness and Startup).
- Connection reuse defeating distribution, so a "balanced" service has a badly skewed load and one backend is the bottleneck.
- Sticky sessions pinning users to a backend, so removing it drops their sessions and a rollout is user-visible (Sticky Sessions in the backend view).
- Deregistration delay shorter than the longest in-flight request, dropping work on every scale-in and every deploy.
- A cold backend added at full share, producing a latency spike attributed to the deploy rather than to warm-up.
- Retries at the load balancer amplifying an overload: every failing request becomes several, and a struggling backend receives more traffic than before (Retry Storms: The Load You Generated Yourself in the backend view).
- The load balancer's own connection limit reached, which fails traffic for every service behind it regardless of backend health.
- "Round-robin means even distribution." It means even selection at whatever granularity the balancer selects. With reuse, that can be very uneven traffic (Service Discovery in Operation).
- "The load balancer provides high availability." It provides distribution and removal. Availability requires enough capacity to survive the removal (Capacity During Failover).
- "Health checks should be thorough." A thorough shared check empties the pool. Depth belongs in readiness with a considered scope, not in the thing that decides whether anyone gets served.
- "Adding a backend adds capacity immediately." It adds capacity as it warms up, and briefly makes latency worse while it does.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Requests per second and error rate per backend, not aggregated — the only view in which skew and a single sick backend are visible.
- Healthy target count over time, compared against the desired count, with dips explained by deploys rather than unexplained.
- Errors during scale-in and deploys at zero, which is the direct test of whether draining is configured correctly.
- Latency of a newly added backend compared to established ones, which shows whether warm-up is a real effect for this workload.
- The load balancer's own saturation signals — connection counts and rejected connections — treated as first-class capacity metrics (Headroom).
- Membership changes are immediately reversible: returning a backend to the pool restores its traffic within a health-check interval.
- Health check configuration changes are reversible and are not instant in effect — a stricter check removes backends immediately, and restoring it takes the success threshold to bring them back.
- Nothing rolls back the requests dropped by a bad drain. Those are gone, which is why the drain configuration is worth verifying before you need it.
- Automate registration, deregistration and health-based removal completely. Manual pool management does not survive contact with autoscaling (Toil).
- Automate the capacity check: alert when healthy backend count falls below what your traffic requires, rather than when it reaches zero (Alert on Symptoms, Not on Causes).
- Keep the health check definition owned by the service team, and keep the decision to remove a whole pool — a maintenance mode, a region drain — human (Region Failover).
- L7 balancing gives per-request distribution, retries and routing, at the cost of terminating connections, more CPU, and a component that understands your protocol and can therefore be wrong about it.
- Aggressive health checks remove bad backends quickly and remove good ones during transient blips; conservative ones keep a broken backend serving errors for longer.
- Least-connections and similar adaptive rules distribute better under uneven request cost and can herd traffic onto a backend that is fast because it is failing quickly.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- CLOUD-SPECIFICHealth-check thresholds, deregistration delay, slow start and connection limits are per-provider settings with different names, different defaults and different semantics. A configuration that drains cleanly on one provider drops requests on another with the "same" settings.
- KUBERNETES-SPECIFICIn a cluster there are usually two balancing layers — an external load balancer to the nodes or ingress, and a node-local dataplane to the pods — each with its own membership and its own delay. That doubles the propagation windows, and it means a pod can be out of the endpoint list while the external balancer still forwards to its node. On a plain VM fleet there is one layer and one list.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.