TCP Retransmits on a Noisy-Neighbour VPS: Measuring the Rate and Fixing the Cause

Pathological TCP retransmission rates on a VPS are almost never a kernel-tuning problem. A healthy wired link retransmits well under 0.1% of segments; 2–8% on a single instance, while the same destination from a laptop is clean, usually means one of three things: a shared host uplink dropping packets during someone else’s burst, an MTU/PMTU black hole on one route, or a socket buffer and queue discipline combination that turns host-side drops into congestion-window collapse. Each has a different fix, and the order matters because the measurement you would reach for first (iperf3) hides all three.

What follows separates retransmits by cause using ss -ti, nstat and a short tcpdump sample, then maps each cause to a fix you can apply without a reboot. The same counters also tell you whether the instance itself is misbehaving — if the numbers are worse than a standard KVM VPS baseline network test suggests they should be, that is a support ticket rather than a sysctl.

Measure the retransmit rate correctly

Two counters matter and they are not the same: TCPRetransSegs (segments retransmitted) and TCPLostRetransmit (retransmissions that were themselves lost). A high TCPLostRetransmit relative to TCPRetransSegs means the path is dropping retransmits too, which points at the uplink rather than at your configuration.

# snapshot counters, wait, snapshot again
nstat -az > /tmp/n1.txt; sleep 300; nstat -az > /tmp/n2.txt
diff /tmp/n1.txt /tmp/n2.txt | grep -iE "retrans|lost|timeout|resets"

# per-connection view: retransmits, RTT, cwnd, unacked
ss -tin state established | head -40

Read the per-connection line as follows: retrans:2/120 means two segments retransmitted out of 120 in flight at peak; lost:1 is the ones lost again; cwnd:14 with ssthresh:7 means you were recently cut back. A rtt of 0.6 ms with rttvar jumping to 40 ms is the signature of queue buildup on the host uplink, not of congestion on the public path.

Establish a denominator

A ratio needs a total. Compute it from nstat deltas over a fixed window rather than eyeballing a percentage in a dashboard.

awk '
  FNR==NR{ if($2=="TcpOutSegs"||$2=="TcpRetransSegs") a[$2]=$3; next }
  $2=="TcpOutSegs"{ out=$3 } $2=="TcpRetransSegs"{ rt=$3 }
  END{ printf "out=%d retrans=%d rate=%.3f%%\n", out-a["TcpOutSegs"], rt-a["TcpRetransSegs"], 100*(rt-a["TcpRetransSegs"])/(out-a["TcpOutSegs"]) }
' /tmp/n1.txt /tmp/n2.txt

Benchmarks to hold yourself to: below 0.05% is clean on a wired path; 0.1–0.5% is tolerable for long-haul but worth a look; above 1% is a defect for typical web traffic. Under 5% packet loss, TCP throughput collapses roughly in proportion to 1/sqrt(p), so 2% loss costs you about a third of your achievable bandwidth.

Separate host-side drops from path loss

Host-side drops never leave the machine, so a packet capture on the public interface will show them as gaps with no corresponding ICMP. Compare the virtio interface counters with what actually went out.

ip -s link show dev eth0
# tx dropped / tx errors / carrier should be flat
ethtool -S eth0 2>/dev/null | grep -iE "drop|miss|discard|tx_queue"

# softnet backlog drops: per-CPU, the quiet killer on small VPS
grep -E "^(smp|net)" /proc/net/softnet_stat | awk '{print "drops:"$2" time_squeeze:"$3}'

The second column of /proc/net/softnet_stat is the per-CPU backlog drop counter. Any nonzero value on a 1–2 vCPU instance means the packet-processing path is saturating: the NIC receives frames faster than softirq can drain them. That produces retransmits with zero loss on the wire, and no amount of tcp_* tuning fixes it — the queues need to be smaller or the receiver needs more CPU budget.

Confirm with a targeted capture

When the counters disagree, one 10-second capture settles it. Filter to a single busy connection so tshark’s analysis is meaningful.

sudo timeout 10 tcpdump -i eth0 -s 96 -w /tmp/rt.pcap \
  'tcp and host 198.51.100.20 and port 443'
tshark -r /tmp/rt.pcap -q -z io,stat,0 \
  -z "io,stat,0,COUNT(tcp.analysis.retransmission)tcp.analysis.retransmission" \
  -z "io,stat,0,COUNT(tcp.analysis.fast_retransmission)tcp.analysis.fast_retransmission" \
  -z "io,stat,0,COUNT(tcp.analysis.lost_segment)tcp.analysis.lost_segment"

Fast retransmissions triggered by three duplicate ACKs with the same sequence numbers repeating every few hundred ms indicate a single persistent drop point. Retransmissions spread evenly with wildly varying sequence gaps indicate congestion, and a capture showing retransmits only for packets above 1400 bytes indicates PMTU. Check the last case with ping -M do -s 1472 198.51.100.20: silence plus a working smaller size confirms it.

Fix 1: smaller buffers on a shared uplink

If queuing delay appears in rttvar and softnet drops are climbing, the fix is to stop buffering so much on the way out. A smaller txqueuelen plus fq_codel keeps latency bounded; tcp_notsent_lowat stops the kernel from holding large unsent windows in memory.

sudo sysctl -w net.ipv4.tcp_notsent_lowat=131072
sudo tc qdisc replace dev eth0 root fq_codel limit 1024
sudo ip link set eth0 txqueuelen 500
tc -s qdisc show dev eth0

Fix 2: repair PMTU black holes

If only full-size segments are lost and ICMP “fragmentation needed” never arrives (common where icmp is filtered), TCP stalls until the retransmit timer fires. Clamp the MSS to the path you can actually reach — the standard fix behind VXLAN, WireGuard and many provider overlays.

# verify the path MTU first (wireguard/wg0 usually needs 1420)
ping -M do -s 1420 -c 2 198.51.100.20
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

Fix 3: escalate when the evidence points upstream

Two findings indicate a host problem rather than a guest configuration problem: retransmits that correlate with hours of the day and other tenants’ behaviour, and tx dropped on the virtio interface rising while softirq is idle. In that state, your packets are being discarded after the guest’s queue and there is nothing in a sysctl that can recover them. Collect nstat deltas, the ip -s link counters and a filtered capture, and hand them over — with numbers, providers respond far faster than to “the network is slow”.

Whatever the outcome, keep the measurement running. A cron job that snapshots nstat -az every five minutes and appends the delta to a TSV gives you a week of history, which is the only reliable way to tell a persistent defect from a twenty-minute neighbour burst. And if the pattern turns out to be sustained host contention rather than path loss, moving to a VPS with dedicated or guaranteed network resources solves it in an afternoon whereas tuning will not.

Leave a Reply