TCP: A Reliable Ordered Byte Stream over an Unreliable Network

TCP promises a connection between two endpoints carrying a reliable, ordered, full-duplex byte stream. Reliable: every byte you write is delivered or the connection fails — there is no third outcome. Ordered: bytes are read in the order they were written. Full duplex: both directions are independent streams. Byte stream: the unit is the byte; the boundaries of your write() calls do not exist on the other side.

It makes this promise over IP, which promises nothing (IP: Best-Effort Delivery Between Machines). The gap is closed by a small set of mechanisms that each answer one failure: sequence numbers so the receiver can order and de-duplicate; acknowledgments so the sender learns what arrived; retransmission on timeout or duplicate ACKs for what did not; flow control so the sender does not overrun the receiver’s buffer; congestion control so it does not overrun the network; and a checksum so corruption is treated as loss. A handshake at the start agrees the numbering, and a close sequence at the end ensures both sides know it is over.

The cost is latency and state. Setting up a connection is a round trip before the first byte (The Three-Way Handshake); a lost segment stalls the stream for at least a round trip and often several (Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO); a new connection starts slowly and ramps up (Congestion Control: Protecting the Network); and both endpoints hold buffers and timers for every open connection (The Connection Lifecycle: Close, Reset, TIME_WAIT, CLOSE_WAIT). TCP is the correct default for anything that must be complete and correct; UDP: Datagrams and the Contract You Choose is the alternative when it is not.

Each mechanism is its own lesson; here is the map. The receiver acknowledges the highest in-order byte it has (cumulative ACK) and, with SACK, the out-of-order ranges too. The sender keeps everything unacknowledged in its buffer and retransmits when a timer fires or when three duplicate ACKs say the receiver has a hole. It never has more than min(rwnd, cwnd) bytes in flight: rwnd, the receiver’s advertised free buffer, protects the receiver; cwnd, the sender’s estimate of what the network will carry, protects the network.

The most common TCP bug in application code has nothing to do with the network. A client does send(json); the server does recv() and parses what it gets. It works in every test because on a LAN a small write usually arrives as one segment and is read in one call. In production the server receives half a JSON document, or one and a half, and the parser throws. TCP never promised that one send is one recv: segments are split at the MSS, merged by Nagle’s algorithm or by the receiver’s buffer, and delivered as "whatever bytes are in order right now".

Every protocol over TCP therefore defines its own framing: a length prefix (gRPC, most binary protocols), a delimiter (HTTP/1.1 headers end at a blank line, then Content-Length or chunked encoding frames the body; Redis uses \r\n), or a fixed-size header that says how much follows. The receiving code must buffer until a complete frame is present and must handle a read that returns several frames at once. If you write a TCP protocol and skip this step, it will fail in exactly the way that cannot be reproduced locally.

A connection is identified by the 4-tuple (source IP, source port, destination IP, destination port) — plus the protocol, which is why some say 5-tuple. Every incoming segment is matched to a socket by that tuple, which is how one server on port 443 serves fifty thousand clients: each has a different source address or port. It is also why a client that opens many connections to one server needs many source ports, the subject of The Connection Lifecycle: Close, Reset, TIME_WAIT, CLOSE_WAIT and Ports: Addressing a Process, Not a Machine.

The header is 20 bytes minimum, up to 60 with options. Ports (2×16 bits), a 32-bit sequence number, a 32-bit acknowledgment number, the header length, flags (SYN, ACK, FIN, RST, PSH, URG, plus ECE/CWR for ECN), a 16-bit window, a checksum, an urgent pointer nobody uses, and options: MSS, window scale, SACK-permitted / SACK blocks, and timestamps. With a 20-byte IPv4 header and a 1500-byte MTU, the MSS is 1460 bytes of payload per segment; timestamps and SACK options reduce it to 1448 in most real segments.

On every mainstream OS, TCP is in the kernel. Your program sees a socket — a file descriptor with a send buffer and a receive buffer behind it (The Socket: A Descriptor With Two Kernel Buffers Behind It, The Buffer Chain). write() copies into the send buffer and returns; the kernel segments, sends, retransmits and paces on its own timers, driven by interrupts from the NIC and by the ACKs that come back. read() returns whatever in-order bytes have accumulated in the receive buffer, or blocks until some do. Nothing about segments, ACKs or windows is visible from the API, which is both the point and the reason the framing bug exists.

Six lines are enough to use it. What they hide — DNS, the handshake, slow start, retransmission, the close — is the rest of this module. User-space TCP stacks exist (in DPDK-based systems, in some databases and load balancers, and QUIC is effectively one over UDP), but "TCP" in an ordinary program means the kernel’s.