The question this answers
If every pod's IP address changes whenever it is replaced, how does anything ever find anything?
The web frontend must call the API. The API has three replicas that are replaced on every deploy, rescheduled on every node failure and scaled up and down by traffic. The frontend cannot be redeployed every time one of those happens.
A stable virtual address and DNS name whose membership is maintained automatically: callers use one name, and the set of pods behind it changes underneath without anyone being told.
The problem is churn, not routing
A pod's IP is assigned when it starts and is gone when it stops. That is not a defect to work around — it is the direct consequence of pods being disposable, which is the property everything else in this module depends on. But it means the address of "the API" is a moving target: three addresses right now, a different three after the next deploy, five after an autoscaling event.
A Service is the indirection that absorbs that churn. It has a virtual IP that never changes for the life of the Service, and a DNS name derived from its own name and namespace, so callers resolve api (or api.production.svc.cluster.local) and get a stable answer. Behind it, a controller maintains the list of pod addresses that currently match the Service's label selector and are Ready. That list is the actual routing table, and it is rewritten continuously.
Two words in that sentence carry the weight. Selector: the Service does not know about your Deployment, it matches labels — so a label typo yields a Service with an empty endpoint list while every pod is perfectly healthy, and the error the caller sees is an immediate connection failure, not a timeout. Ready: only pods passing their readiness probe are in the list, which is exactly the mechanism that makes rolling updates safe and exactly the mechanism a badly written probe defeats.
What actually happens to the packet
The virtual IP is worth understanding because it is not a machine. Nothing is listening on 10.96.4.20. It is a rule programmed onto every node in the cluster: traffic destined for that address is rewritten, on the sending node, to the address of one of the currently-Ready pods. There is no proxy hop, no extra network round trip and no central component in the path — which is why Service routing survives a control-plane outage perfectly well, while *changes* to the membership list do not.
The load balancing is per-connection and effectively random, not per-request and not smart. For HTTP/1.1 with short connections that approximates round-robin adequately. For long-lived connections — HTTP/2, gRPC, database pools, WebSockets — it does not, because the choice is made once at connect time and then every subsequent request on that connection goes to the same pod. This is the classic gRPC-on-Kubernetes surprise: you scale to ten replicas and traffic stays pinned to the three the clients happened to connect to. The fix is a client-side or L7-aware balancer, not more replicas.
DNS deserves one warning. Cluster DNS is a real service with real caching, and clients that resolve once and cache forever will hold a stale address across a Service recreation. Applications that reconnect should re-resolve; JVM defaults have historically been a repeat offender here.
Which kind of Service, and when
There are four shapes, and choosing wrongly is a common and expensive mistake. The default, a cluster-internal virtual IP, is correct for essentially all internal traffic. A node-port variant exposes the Service on a port of every node, which is mostly a building block rather than something to use directly. A load-balancer type asks the provider to create a real, billed external load balancer — and this is the expensive one, because it is one load balancer *per Service*. A dozen services exposed this way is a dozen load balancers on the invoice and a dozen public IPs on your attack surface.
The usual right answer for external HTTP is: one load balancer at the edge, an ingress or gateway behind it, and internal Services underneath — see Ingress and Gateway: Getting Traffic In. That collapses N load balancers into one and puts routing, TLS and host/path rules in a single place.
The fourth shape, a headless Service, does no load balancing at all: DNS returns every pod address individually. That is what stateful workloads use when a client must address a *specific* replica — the primary of a database, a particular shard — and it is the discovery mechanism Stateful Workloads: Databases Are Not Stateless APIs depends on.
| Type | What it gives you | What it provisions | Use it when |
|---|---|---|---|
| ClusterIP (default) | A stable internal virtual IP and DNS name. | Nothing external. A dataplane rule on each node. | Essentially all internal service-to-service traffic. |
| NodePort | The same, plus a fixed port on every node. | An open port cluster-wide. | Rarely directly — mostly as a building block under a load balancer. |
| LoadBalancer | An external address reachable from outside the cluster. | A real provider load balancer and public IP, per Service, billed hourly. | One or two edge entry points — not per microservice. |
Headless (clusterIP: None) | DNS returning every pod address; no balancing. | Nothing. | When a client must reach a specific replica: databases, shards, peer discovery. |
| ExternalName | A DNS alias to something outside the cluster. | Nothing. | Pointing at a managed database or third-party endpoint by cluster-local name. |
Key points
- Pod IPs churn by design; a Service is the stable name and virtual IP that absorbs the churn.
- Membership is by label selector and readiness — a label typo produces an empty Service behind perfectly healthy pods.
- The virtual IP is a rule programmed on every node, not a host, so there is no extra hop and routing survives a control-plane outage.
- Balancing is per connection, so long-lived connections (gRPC, HTTP/2, pooled database clients) pin to a pod and scaling out does not rebalance them.
- A LoadBalancer-type Service provisions a real, billed load balancer per Service; use an ingress or gateway instead of one per microservice.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A Service is created with a label selector and a stable virtual IP allocated from the cluster service range.
- • A controller continuously computes the set of pods matching the selector that are also Ready, and publishes it as the Service's endpoints.
- • Every node's dataplane programs rules that rewrite traffic for the virtual IP to one of those endpoint addresses.
- • Cluster DNS resolves the Service name to the virtual IP, so callers never learn a pod address at all.
- • On every pod change the endpoint list is rewritten and the node rules are updated — the propagation is fast but not instantaneous.
- • Label hygiene, because selectors are the wiring and a rename is a routing change with no compile-time check.
- • Readiness probe correctness, since it decides membership: too lax admits broken pods, too strict empties the Service.
- • Client-side connection behaviour for long-lived protocols — this is application work that no Service setting can do for you.
- • DNS caching behaviour in your runtimes, and re-resolution on reconnect.
- • The cost and exposure of every LoadBalancer-type Service, each of which is a real internet-facing resource.
- • Selector mismatch: zero endpoints, immediate connection refused, and every pod reporting healthy. The most common Service bug there is.
- • All pods failing readiness: the Service empties and the caller sees a hard failure rather than a slow one — arguably better, definitely surprising.
- • Connection pinning: after scaling from three to ten replicas, three pods are saturated and seven are idle, because existing connections never moved.
- • Stale DNS in a client that cached a resolution across a Service recreation, producing traffic to an address that no longer routes.
- • Endpoint propagation lag during a rollout: a terminating pod still receives a few connections after it began shutting down — which is why graceful shutdown exists.
- • Endpoint churn, not request volume, is what pressures the control plane; a large fleet rolling out rewrites large endpoint lists repeatedly.
- • Very large endpoint counts per Service stress the dataplane on every node, which is why endpoint slicing exists in modern versions.
- • Per-connection balancing means throughput scales with replicas only for short-lived connections; for long-lived ones you must rebalance clients.
- • A ClusterIP Service is reachable from every pod in the cluster by default — the cluster network is flat unless you add network policy.
- • That flatness is the argument for network policy as a real trust boundary between namespaces, not just a compliance checkbox — see Infrastructure Trust Boundaries.
- • A LoadBalancer Service is a genuine public exposure decision. A public load balancer on 443 in front of an API is the design; one in front of a database port is the finding — see Public Exposure, Read With Context.
- • Service names are predictable and DNS is cluster-wide, so internal service discovery is not a secret and must not be treated as an access control.
- • ClusterIP Services are free — they are rules, not resources.
- • LoadBalancer Services are the expensive shape: an hourly charge and a data-processing charge, per Service, forever.
- • Cross-zone traffic between a Service and its pods can be billed as inter-zone data transfer, which is a real and frequently invisible line item at scale.
- • Endpoint count per Service — the fastest check that anything is behind the name at all.
- • Per-pod request distribution, which is where connection pinning becomes visible as a wildly uneven split.
- • Cluster DNS latency and error rate; DNS problems present as broad, confusing, intermittent failures across unrelated services.
- • The signal that lies: pod health. Every pod can be healthy while the Service routes to none of them, because health and selector matching are unrelated.
- • An external DNS name and a normal load balancer, if the caller is outside the cluster anyway — you do not need cluster service discovery for that.
- • A headless Service plus client-side load balancing, which is the right answer for gRPC and for anything needing per-request distribution.
- • A service registry your applications already use, if you have one — running two discovery systems is worse than either.
- • On a single VM,
localhostand a port. The entire problem this solves is created by pods moving.
- • Buys stable naming and automatic membership; costs a routing layer wired by label strings with no type checking.
- • Buys a hop-free dataplane; costs per-connection rather than per-request balancing, which quietly breaks for modern protocols.
- • Buys instant reachability between everything in the cluster; costs a flat network that you must then deliberately segment.
What people believe, and what is true
A Service load balances requests.
It load balances connections. Every request on one HTTP/2 or gRPC connection lands on the same pod until that connection closes.
The ClusterIP is a proxy the traffic passes through.
It is a rule on the sending node. There is no host at that address and no extra network hop.
A Service knows which pods belong to its Deployment.
It matches labels. Any pod with those labels joins the Service, including one from a different workload that happens to share a label.