ScalingGENERALCLOUD-SPECIFICPROTOCOL-SPECIFIC
Load Balancing, From the Backend's Side
What your application owes the thing distributing traffic to it — an honest health signal, aligned timeouts, and no assumption about which instance gets what.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The question
What does my backend have to do to be distributed across correctly, and what can the load balancer not fix?
The requirement
Traffic must be spread across instances so that no instance is overloaded, an unhealthy instance stops receiving requests, and a deploy does not drop any.
The obvious build
Put the instances behind a load balancer with round-robin and a health check on /health. Distribution is the load balancer's job from there.
Why it breaks
One instance is much slower than the others — a slow disk, a noisy neighbour, a stuck garbage collection — and round-robin keeps sending it an equal share, so a fraction of all requests are slow (Tail Latency: Why p50 Being Fine Does Not Help in Observability & Performance).
How it breaks in production
- One instance is much slower than the others — a slow disk, a noisy neighbour, a stuck garbage collection — and round-robin keeps sending it an equal share, so a fraction of all requests are slow (Tail Latency: Why p50 Being Fine Does Not Help in Observability & Performance).
/healthreturns 200 while the database connection is dead, so a broken instance keeps taking its share and returning errors (Health Checks: Startup, Readiness, Liveness).- Long-lived connections (HTTP/2, gRPC, WebSockets) pin a client to one instance for hours, so a fleet that is balanced at connection time becomes badly imbalanced as traffic patterns shift.
- The load balancer's idle timeout is shorter than the application's keep-alive timeout, so it forwards a request onto a connection the backend is closing, and the client sees a 502 with nothing in the application logs.
- The load balancer retries a failed request on another instance — including a POST that had already been processed — and the effect happens twice (Idempotency in Backends).
- New instances receive full traffic the instant they pass readiness, on a cold pool and an empty cache, so every scale-up produces a latency spike (Autoscaling a Backend).
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale
What is actually happening
- A load balancer picks a backend per connection (layer 4) or per request (layer 7). That distinction decides almost everything else: an L4 balancer cannot rebalance a long-lived connection, and an L7 balancer can route each request independently.
- Selection algorithms differ in what they know. Round-robin knows only order. Least-connections knows in-flight counts and therefore adapts to slow instances. Power of two choices samples two at random and picks the less loaded, getting most of the benefit at a fraction of the coordination. Hashing routes deterministically by a key, which is how affinity and cache locality are implemented.
- Health checking is the mechanism that removes an instance from rotation. It is only as good as the endpoint: a check that verifies nothing removes nothing (Health Checks: Startup, Readiness, Liveness).
- Connection reuse between the balancer and the backend matters as much as between client and balancer. Mismatched keep-alive settings cause requests to be sent onto connections that are being closed — a race with no application-level explanation (Keep-Alive and Connection Reuse).
- Timeouts exist at every hop. The shortest one wins, and the application's timeout is decoration if the balancer gives up first (Timeouts).
- Retries at the balancer are a different policy from retries in the client, and both can be active. A request retried at two layers is attempted a multiple of times (Retry Storms).
- Draining is the balancer's side of graceful shutdown: it stops sending new requests to an instance and allows in-flight ones to finish. Deregistration is asynchronous, which is why the application still needs a drain delay (Graceful Shutdown).
Algorithms differ in what they know
Every selection algorithm is a trade between how much it knows about instance state and how much coordination that knowledge costs. Round-robin knows nothing and costs nothing. Least-connections knows in-flight counts and adapts to a slow instance automatically. Power of two choices gets most of that adaptivity by sampling two instances at random, which is why it is a common default in modern proxies.
The practical question is whether your requests and instances are homogeneous. If some requests cost a hundred times more than others, or if one instance can be transiently slow, an algorithm that only counts requests will send the slow instance exactly as much work as the healthy ones.
| Algorithm | What it knows | Good when | Fails when |
|---|---|---|---|
| Round-robin | Order only | Uniform instances, uniform request cost | One instance is degraded — it keeps getting an equal share. |
| Least connections | In-flight count per instance | Variable request cost; slow instances shed load automatically | Long-lived connections make the count meaningless. |
| Power of two choices | Two sampled instances | Large fleets where global state is expensive | Very small fleets, where sampling adds little. |
| Weighted | Static per-instance capacity | Heterogeneous instance sizes | Weights drift out of date as the fleet changes. |
| Hash / consistent hash | A key from the request | Cache locality, affinity, partitioned state | Key skew concentrates a large tenant on one instance (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Observability & Performance). |
| Random | Nothing | Simple and surprisingly acceptable at scale | Same weakness as round-robin, with more variance. |
The timeout ladder, and the keep-alive race
RUNTIME-SPECIFICNode exposes
keepAliveTimeout and headersTimeout explicitly and defaults them low enough to hit this problem behind a managed balancer. Go's http.Server uses IdleTimeout; a JVM container has its own equivalent. The rule — backend idle timeout strictly greater than the proxy's — is universal.Two configuration mismatches account for most unexplained 502s in front of a healthy service. The first is timeout inversion: the balancer gives up before the application does, so the application keeps working — holding a database connection — for a client that has already received an error.
The second is subtler. Both the balancer and the backend have an idle timeout on their pooled connections. If the backend closes first, there is a window where the balancer sends a request onto a connection that is already closing. The result is a failure the application never sees, at a rate proportional to how often connections sit idle. The fix is to make the backend's idle timeout comfortably longer, so the balancer is always the side that closes.
Aligning the ladder from the application side
1// The ladder, outermost first. Each layer must be2// strictly shorter than the one outside it.3//4// client timeout 30s5// > LB request timeout 25s6// > app request timeout 20s7// > outbound call cap 15s (leaves room to respond)8//9// If the app is not the shortest, it does work for a10// client that has already given up — holding a pool11// connection the whole time.12 13const server = http.createServer(app)14 15// LB idle timeout is 60s (a fixed platform value).16// The backend MUST outlive it, or the LB will dispatch17// onto a socket we are closing -> 502 with no app log.18server.keepAliveTimeout = 75_000 // > 60s19server.headersTimeout = 80_000 // > keepAliveTimeout20 21// Per-request deadline, propagated to everything downstream22app.use((req, res, next) => {23 const ac = new AbortController()24 const t = setTimeout(() => ac.abort(), 20_000)25 res.on('close', () => clearTimeout(t))26 req.signal = ac.signal // handlers pass this to db + http calls27 next()28})The keepAliveTimeout > LB idle timeout rule is the one that is almost never written down and causes a steady background rate of 502s that no application log explains. The direction matters: the side that closes an idle connection should be the one that is not about to send a request on it.
What the balancer cannot fix
A load balancer distributes requests. It cannot make an instance healthy, cannot know what a request will cost, and cannot make a non-idempotent operation safe to retry. Every failure below looks like a balancing problem and is resolved in the application.
The most valuable habit is to check the per-instance view before blaming the balancer: equal request counts with unequal latency is a sick instance and a health check that did not notice.
Symptoms that look like the load balancer
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| One instance degrades | A fraction of all requests are slow; averages look fine | Round-robin keeps sending it an equal share | Least-connections or outlier ejection; a readiness check that detects the degradation (Tail Latency: Why p50 Being Fine Does Not Help in Observability & Performance). |
| Idle traffic period | Sporadic 502s with no application log line | Backend keep-alive idle timeout shorter than the balancer's | Raise the backend idle timeout above the balancer's. |
| Deploy | Burst of connection resets during rollout | Deregistration is asynchronous; the instance stopped accepting too early | Drain delay in the application (Graceful Shutdown). |
| Scale-up | Latency spike right after new instances appear | Cold pools and caches taking full traffic immediately | Slow start at the balancer, or warm before signalling readiness. |
| Dependency slowdown | Every instance removed at once; total outage | Readiness checks a shared dependency, so all fail together | Minimum healthy fraction, and readiness that degrades rather than fails where the service can still serve some traffic. |
| gRPC / HTTP/2 clients | Two instances at 90% while four idle | Connection-level balancing pinning multiplexed streams | L7 balancing, client-side balancing, or a maximum connection lifetime to force reconnection. |
| Partial failure | Duplicate side effects after a transient error | Balancer retried a non-idempotent request on another instance | Restrict retries to safe methods; use idempotency keys (Idempotency Keys). |
How to build it
Most important first.
- Make readiness mean "this instance can serve a real request": dependencies reachable, config valid, pool built, and failing while draining. Everything else about balancing depends on that signal being honest.
- Prefer least-connections or power-of-two-choices over strict round-robin for services with variable request cost. Round-robin is correct only when every instance is equally capable and every request is equally expensive.
- Align the timeout ladder deliberately: client timeout > balancer timeout > application timeout, and set the application's keep-alive idle timeout *longer* than the balancer's so the balancer is always the side that closes.
- Decide retry policy in exactly one place, and only for requests that are safe to repeat. "Retryable" is a property of the error; "safe to retry" is a property of the operation, and they are not the same (Retries).
- For long-lived connections, add a maximum connection lifetime so clients periodically reconnect and the fleet can rebalance. Without it, an L4 balancer in front of gRPC or HTTP/2 will stay imbalanced indefinitely.
- Ramp new instances rather than exposing them to full traffic instantly — slow start at the balancer, or a warmup step before readiness passes.
- Spread instances across failure domains and let the balancer route around a zone rather than an instance (Multi-Zone Deployment in Cloud & Infrastructure).
- Shed load rather than queue it unboundedly: an instance that returns 503 quickly when saturated lets the balancer route elsewhere, while one that accepts everything becomes a queue (Backpressure).
What can go wrong
Failure modes
- A health check so shallow that it never fails, or so deep that a dependency blip removes every instance at once — both are common, and the second is worse.
- Idle-timeout mismatch producing 502s that appear at the balancer and never in the application logs.
- Retry amplification: the client retries, the balancer retries, and a struggling backend receives several times the offered load exactly when it can least handle it (Retry Storms).
- Connection-level balancing in front of multiplexed protocols, giving a fleet whose load distribution was decided by whoever connected first.
- Cross-zone routing turning a local call into a cross-zone one, adding latency and, on some platforms, a data-transfer cost.
- A hash-based algorithm concentrating traffic when the key is skewed — one large tenant hashing to one instance (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Observability & Performance).
What can race
- A request can be routed to an instance in the same moment the instance is being deregistered, arriving after it has begun shutting down (Graceful Shutdown).
- The balancer can dispatch a request onto a keep-alive connection the backend has decided to close, producing a failure attributable to neither side alone.
- A retried request can execute concurrently with the original if the first attempt is still running behind a timeout — two executions of one intent (Duplicate Detection).
Security
- TLS usually terminates at the balancer. Know whether the hop from balancer to instance is encrypted, and do not describe the system as end-to-end encrypted if it is not (TLS as a Security Boundary in Security Engineering).
- Client IP arrives in a forwarded header, which is client-controllable unless the balancer overwrites it and the application only trusts the value from a known proxy. Getting this wrong silently breaks IP-based rate limiting (Rate Limiting).
- Instances should be reachable only through the balancer. An instance with a public address bypasses every policy the balancer enforces (Public Exposure, Read With Context in Cloud & Infrastructure).
- Health endpoints should not expose internal state — dependency hostnames, versions, configuration — since they are often the least protected route in the service (Not Leaking Your Internals).
Misreads
- "The load balancer distributes load." It distributes *requests* or *connections*. Load is what those requests cost, and no balancer knows that in advance.
- "Round-robin is fair." It is uniform, which is fair only when instances and requests are homogeneous. Under heterogeneity it sends equal traffic to unequal servers.
- "Health checks make the service self-healing." They remove instances that fail a check. If the check does not test what is broken, nothing is removed (Health Checks: Startup, Readiness, Liveness).
- "The load balancer will handle the deploy." It handles routing; the instance still has to drain. Deregistration is asynchronous, so both sides participate (Graceful Shutdown).
- "We are balanced because the request counts are equal." Equal counts with unequal latency means one instance is degraded and taking its full share of traffic.
Operating it
How you see it in production
- Compare per-instance request rate, error rate and latency. Balanced traffic with unbalanced latency means one instance is unwell; unbalanced traffic means the algorithm or the connection model is not doing what you think.
- Track the gap between latency measured at the balancer and inside the handler. A widening gap is queueing ahead of your code (Queueing: Why Systems Get Slow Before They Get Broken in Observability & Performance).
- Alert on healthy-target count, not only on error rate. A fleet quietly running at half capacity is invisible in request metrics until it is not.
- Count balancer-level retries separately from client retries so amplification is measurable rather than inferred.
What changes at 10x and 100x
- At small instance counts, algorithm choice barely matters. At large ones, the difference between round-robin and least-connections shows up directly in the tail.
- At high connection rates, connection reuse between balancer and backend dominates; without it the fleet spends its capacity on handshakes (Keep-Alive and Connection Reuse).
- At very large scale, one balancer tier is not enough and the problem becomes hierarchical — DNS or anycast in front of regional balancers in front of instance pools (Load Balancers as Infrastructure in Cloud & Infrastructure).
What this costs
- Smarter algorithms need per-instance state and add coordination; power-of-two-choices exists precisely because perfect knowledge is expensive.
- Deeper health checks detect real problems and risk correlated removal of the whole fleet when a shared dependency degrades. The mitigation — a minimum healthy fraction — is itself a policy decision.
- Balancer-level retries improve success rates for idempotent traffic and amplify load during an incident. It is a real trade, not a free win.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALSelection algorithms, health checking and the timeout ladder behave the same across implementations.
- CLOUD-SPECIFICManaged balancers impose idle and request timeouts you cannot exceed from inside the application, and differ in whether they support slow start, per-request retries and outlier ejection. Cross-zone routing behaviour also differs, sometimes with a data-transfer cost attached.
- PROTOCOL-SPECIFICHTTP/1.1 gives one request per connection at a time, so per-connection balancing approximates per-request balancing. HTTP/2 and gRPC multiplex many requests over one long-lived connection, so an L4 balancer pins all of them to one backend and the fleet stays imbalanced until connections are recycled.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
API Designretries-and-timeouts