Service Discovery

With three fixed servers, orders.internal = [10.0.1.4, 10.0.1.5, 10.0.1.6] in a config file is fine, and it stays fine for years. It breaks on the day the fleet becomes dynamic. An autoscaler adds two instances at 9 a.m. that no config mentions, so they receive no traffic and the scale-out achieves nothing. A rolling deploy replaces every instance with a new one at a new address, so the old list is 100% wrong for the duration of the deploy. A container scheduler moves a crashed instance to another host with a different IP and port. In each case the truth about "where is the order service" changes many times a day, and something must publish that truth to whoever needs it. That something is the service registry: a small, highly available database of service → [instance address, port, health, metadata], kept current by the instances themselves or by the platform that runs them.

In client-side discovery, the caller queries the registry, caches the instance list, picks one (round robin, least loaded, ring-hash for affinity as in Consistent Hashing) and calls it directly. One hop, full control over balancing, and the client library must be written and kept current in every language the fleet uses. In server-side discovery, the caller sends to a stable name — a load balancer, a Kubernetes Service virtual IP, a gateway — and *that* component queries the registry and forwards. The client is trivial and language-agnostic; the cost is an extra hop and a component that must itself be highly available. Most platforms today are server-side by default (Kubernetes Service + kube-proxy, cloud load balancers with target groups) with client-side used where affinity or latency matters.

Instances get into the registry by self-registration (the process registers on start and deregisters on shutdown) or by third-party registration, where the orchestrator that created the instance registers it — Kubernetes populates Endpoints from pod readiness, so the application never talks to the registry at all. Self-registration is simpler to reason about and fails when the process dies without running its shutdown hook — kill -9, OOM, a host that loses power. Hence health: either the registry actively probes each instance (an HTTP GET /health every few seconds, with a threshold of consecutive failures) or each instance sends a heartbeat that renews a TTL, and an entry whose TTL expires is removed.

The TTL is the number that decides how long dead instances receive traffic. With a heartbeat every 10 s and a TTL of 30 s, a crashed instance is served to callers for up to 30 s. With a TTL of 5 minutes — the setting in the challenge traffic-to-dead-instances — an autoscaler that terminates a third of the fleet leaves a third of all calls failing for five minutes, on every scale-in, every day. Shorter TTLs mean more heartbeat load on the registry and more false removals under GC pauses or network blips; the usual compromise is 10–30 s, plus passive health: the caller marks an instance bad after a connection error and skips it, without waiting for the registry at all. Deregistration on graceful shutdown should come first: deregister, wait for in-flight requests to drain, then exit — the rolling-deploy path in Stateless vs Stateful Services.

DNS is the oldest registry, and often enough. A name resolves to the current set of healthy addresses (A records, or SRV records that carry ports too); Kubernetes headless services and Consul both serve discovery this way. The trap is caching: resolvers, JVMs and HTTP client libraries cache DNS answers, sometimes ignoring the TTL, so a caller can hold a dead address for minutes. Set short TTLs (5–30 s), make the client library re-resolve, and treat a DNS answer as a hint that passive health checking must confirm. DNS also cannot express weights, zones or draining state, which is where a proper registry API earns its place.

A service mesh moves the whole problem out of the application. Every instance gets a sidecar proxy (Envoy) on loopback; the application calls localhost, and the sidecar — fed by a control plane over xDS with the live endpoint list — does discovery, balancing, retries, timeouts, circuit breaking, mTLS and telemetry uniformly for every language. The cost is a proxy per instance (memory, a millisecond of latency), a control plane to run, and a large new thing to understand when something breaks. It pays off for large polyglot fleets that would otherwise reimplement Reliability Patterns in five languages; it is overkill for ten services in one language behind a load balancer.

The registry is a dependency on the critical path of every call, which means it must be more available than anything that uses it. Consul, etcd and ZooKeeper achieve this with a Raft/ZAB quorum of 3 or 5 nodes and by being tiny; callers achieve it by caching the last known good list and continuing to use it when the registry is unreachable — a registry outage should degrade to "no new instances are discovered", never to "no calls succeed". The second failure is staleness in the other direction: the registry says an instance is healthy because its heartbeat arrived, but the instance is deadlocked, out of connections, or has a full disk; a heartbeat proves the heartbeat thread is alive, not that the service works. Health checks should exercise the real path (a lightweight query, not return 200), and passive checks at the caller catch what the registry misses. The third failure is the thundering herd on the registry: 2,000 instances re-resolving every second is 2,000 requests/s that a three-node quorum was not sized for; use watches/long-polling or push (xDS) so callers are notified of changes rather than polling for them.