TCPAdvanced

A 1 Gbit/s link, and the backup crawls at 9 Mbit/s

Symptoms

  • A nightly 40 GB backup between two data centres used to take 25 minutes and now takes over 9 hours.
  • iperf3 between the hosts shows ~9 Mbit/s on a link provisioned for 1 Gbit/s.
  • Both servers are idle: CPU under 5%, disks idle, no memory pressure.
  • Small requests (health checks, SSH sessions) feel fine.
$ ping -c 200 -i 0.2 10.20.0.5 | tail -2
200 packets transmitted, 194 received, 3% packet loss, time 40s
rtt min/avg/max/mdev = 84.1/85.3/119.8/3.2 ms

$ ss -tin dst 10.20.0.5
ESTAB 0 3145728 10.10.0.5:41822 10.20.0.5:5201
    cubic wscale:7,7 rto:288 rtt:85.4/2.1 mss:1448 cwnd:14 ssthresh:12
    bytes_sent:1.9G bytes_retrans:58M retrans:0/41210 lost:0
    delivery_rate:1.1Mbps busy:1640s send 1.9Mbps pacing_rate 2.3Mbps

$ iperf3 -c 10.20.0.5 -t 20 | tail -3
[ ID] Interval  Transfer   Bitrate    Retr
[  5] 0-20 sec  21.4 MBytes 8.98 Mbits/sec  1132   sender
[  5] 0-20 sec  21.1 MBytes 8.85 Mbits/sec        receiver

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

Packet loss on the path
The socket’s TCP state
Receiver window / application read speed
Server CPU, disk, memory
MTU
Where the loss is