Follow One Packet

The interactive asks for four things: a client address (say 192.168.1.42 behind a home router), a server address (203.0.113.10), a protocol (TCP or UDP) and a destination port (443). Everything that follows is determined by those choices plus the state of the devices on the way. This lesson is the narration: at each stop, which headers exist, which fields change, and what the device is doing. The model is conceptual — a real path has more hops, and the numbers are illustrative — but every field behaviour described here is what happens on a real network.

The layers that build the packet are the subject of Encapsulation: Data, Segment, Packet, Frame; the Packet Inspector: Read a Frame Field by Field shows the bytes. Here the focus is movement: the same packet, re-examined at every hop.

The packet is three nested envelopes. The link-layer frame is rewritten completely at every hop: the source MAC becomes the forwarding router’s egress interface and the destination MAC becomes the next hop’s, because MACs only mean something within one link (MAC Addresses: Identity for One Hop). The IP header is mostly preserved end to end — the destination address never changes, and the source changes only at a NAT — but TTL decrements by one per hop and the IPv4 header checksum is recomputed each time because TTL changed (IPv6 has no header checksum, one reason it forwards cheaper). The TCP or UDP header and payload are untouched by ordinary routers; the only middlebox that edits them is NAT, which must rewrite the source port and the transport checksum because the checksum covers a pseudo-header containing the IP addresses.

So at a router in the middle of the internet, the packet still says "from 198.51.100.77:40012 to 203.0.113.10:443, TCP seq 3948211", exactly as the NAT sent it; only the MACs and the TTL reveal how far it has travelled. This is why the destination can reply — the addresses it sees are routable — and why the home router can deliver the reply to the right laptop: the mapping 198.51.100.77:40012 ↔ 192.168.1.42:51742 it recorded on the way out.

A router receives the frame on an ingress port, checks the frame’s checksum and that the destination MAC is its own, strips the frame, and looks at the IP header. It verifies the header checksum, decrements TTL (if it becomes 0: discard, send ICMP Time Exceeded — the traceroute: Discovering the Path Hop by Hop mechanism), and looks the destination address up in the forwarding table with longest-prefix match: among all prefixes that contain 203.0.113.10, the most specific wins (The Routing Table and Longest-Prefix Match; a Trie is the classic structure, and hardware routers use TCAM or compressed tries to do it in a fixed number of memory accesses). The result is an egress interface and a next-hop address. It recomputes the IP checksum, resolves the next hop’s MAC (usually already cached), builds a new frame, and places it in the egress port’s queue.

Time budget: on a 10 Gbit/s port, a 1500-byte packet takes 1.2 µs to serialise, so the lookup and rewrite have to fit in about that long to keep up with line rate — which is why they are done in hardware. Queueing is where variable latency comes from: if the egress port is busy, the packet waits, and under sustained overload the queue fills and the router drops — the loss that Congestion Control: Protecting the Network reacts to. Bufferbloat is the opposite failure: queues so deep the packet waits tens of milliseconds instead of being dropped.

What a router does not do: read ports, sequence numbers or payload (a stateless router forwards on the IP destination alone), reassemble fragments, or keep per-connection state. Anything that does those things — NAT, a stateful firewall, a load balancer — is a middlebox, and middleboxes are where most surprising packet rewrites happen.

The server’s reply carries dst 198.51.100.77:40012 — the NAT’s public address — and travels an independently routed path back. At the home router the mapping turns it into dst 192.168.1.42:51742 and the laptop’s kernel delivers it to the socket whose 4-tuple matches. No device on the return path knows the laptop’s private address ever existed. Switch the lab to UDP and the transport header shrinks to 8 bytes with no sequence numbers, so the destination can no longer tell whether datagrams were lost or reordered — that is the UDP: Datagrams and the Contract You Choose contract. Switch to a destination in the same subnet and the router disappears: ARP resolves the server’s MAC directly and the frame is delivered by the switch with no TTL decrement at all.

This is also the "send a packet" journey seen from the packet’s point of view rather than the OS’s; together they cover both halves of the question "what happens between send() and recv()".