TCP Receive Windows and the Bandwidth-Delay Product on a High-Latency VPS

A 1 GB file transferred across a 200 ms path with default Linux settings moves at roughly 3 MB/s on a 1 Gbps port; the same transfer with BBR and a 4× larger socket buffer reaches 8–10 MB/s. The bottleneck is not bandwidth, it is the product of round-trip time and receive window. This article measures that difference properly — with separate runs for idle RTT, loaded RTT, single-stream throughput and retransmit rate — so you can attribute a slow transfer to the kernel rather than guessing at the network.

Everything here is TCP behaviour inside the guest and on the path; none of it requires host access. If you are still deciding whether a plan gives you enough isolated network capacity to make this worth doing, the cloud VPS scaling model covers how port speed and transfer caps are usually bundled.

Measure the three numbers before tuning anything

Any transfer complaint decomposes into latency, window and loss. Measure them separately, in that order, because a window fix does nothing for a loss problem and a congestion-control fix does nothing for a latency problem.

# 1. Path and RTT, 200 samples
mtr -rwzc 200 -i 0.05 target.example
ping -c 200 -i 0.02 target.example | tail -2

# 2. Single-stream throughput for 20s
iperf3 -c target.example -t 20 -P 1 --json > /tmp/single.json

# 3. Retransmits and window during a transfer
ss -tinmi dst target.example | head -40
nstat -az TcpRetransSegs TcpExtTCPLostRetransmit TcpExtTCPTimeouts

Read iperf3‘s sender and receiver reports separately. A large gap between sender bitrate and receiver bitrate with a retransmit count above a few hundred over 20 seconds points at loss, not at window size. If they match and the rate is far below port speed, the window is the constraint — which is the common case on high-RTT paths.

The bandwidth-delay product is the real limit

Throughput for a single stream cannot exceed window divided by RTT. Linux autotunes receive buffers up to net.ipv4.tcp_rmem‘s maximum, which on most distros defaults to about 6 MB. At 200 ms RTT, 6 MB / 0.2 s = 30 MB/s theoretical — and with loss-driven congestion control on a lossy path the achieved rate is far lower.

RTTWindow needed for 100 MbitNeeded for 1 GbitDefault max 6 MB gives
10 ms~125 KB~1.25 MBplenty
50 ms~625 KB~6.25 MB~960 Mbit
100 ms~1.25 MB~12.5 MB~480 Mbit
200 ms~2.5 MB~25 MB~240 Mbit

Those are clean-path figures. Add 0.1 % loss and CUBIC’s throughput collapses to roughly MSS / (RTT × sqrt(p)), which at 1448 bytes, 200 ms and 0.001 gives roughly 1.1 MB/s — matching the 3 MB/s observation at the top of this article. BBR does not treat loss as a congestion signal in the same way, which is why it recovers most of that loss on a high-RTT path.

Tune the buffer ceiling, not the default

The autotuning defaults matter less than the maximum. Set the ceiling high enough for your worst-case RTT times your port speed, and let autotuning fill it only when the path proves clean:

# /etc/sysctl.d/70-vps-tcp-window.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 262144 16777216
net.ipv4.tcp_wmem = 4096 262144 16777216
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1

Apply with sysctl --system and verify with sysctl net.ipv4.tcp_rmem. Then re-run the single-stream test — this is where people stop, see no change, and conclude buffers do not matter. The measurement in ss explains why: the window was probably never the limit if the path had loss. Look for cwnd: and the rtt: value in ss -tinmi output mid-transfer; if cwnd stays small while retrans climbs, buffers are irrelevant and the path is the problem.

Congestion control: measure the switch, do not assume it

BBR is not universally better. On a low-RTT, lossless LAN or a well-provisioned same-region path, BBR and CUBIC land within a few percent of each other. On high-RTT or lossy paths BBR typically wins by 2–4×. Test both on your actual path rather than trusting a blog post — including this one:

sysctl -w net.ipv4.tcp_congestion_control=cubic
iperf3 -c target.example -t 30 -P 1 | tail -5

sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr
iperf3 -c target.example -t 30 -P 1 | tail -5
sysctl net.ipv4.tcp_available_congestion_control

Run each three times and compare medians, not single runs; the variance between two consecutive iperf3 runs on the same path is often 15 %. If tcp_available_congestion_control does not list bbr, the module is missing: modprobe tcp_bbr and add it to /etc/modules-load.d/.

Separating your bufferbloat from the path’s

The most common self-inflicted problem on a VPS is an oversized egress queue: uploads make interactive traffic unusable because 2 MB of your own frames sit ahead of a 60-byte SSH packet. Measure it by comparing RTT at idle and during a saturating upload.

ip -s link show dev eth0
tc qdisc show dev eth0
# idle
ping -c 100 -i 0.05 target.example | tail -2
# loaded: saturate upstream for 30s, ping meanwhile
iperf3 -c target.example -t 30 -R & sleep 2
ping -c 100 -i 0.05 target.example | tail -2; wait

If p99 RTT rises from 20 ms to 400 ms during the upload, the queue is buffering far too much. Switching egress to fq_codel (which is the default qdisc on many distros but not all) cuts that to typically 30–60 ms with no throughput loss: tc qdisc replace dev eth0 root fq_codel. If a shaping rate is applied by the provider, the queue you see may not be yours at all — test with -R (receive) as well and compare, because downstream shaping is invisible from inside the guest.

Retransmits: rate versus count

TcpRetransSegs is a counter, and counters only mean something as a rate. A busy server retransmits thousands of segments an hour and is perfectly healthy; the useful metric is retransmits as a percentage of segments sent, sampled over a window:

snap() { awk '/TcpOutSegs|TcpRetransSegs/{print $1,$2}' /proc/net/snmp | tail -2; }
snap > /tmp/a; sleep 300; snap > /tmp/b
paste /tmp/a /tmp/b | awk '{d=$4-$2; if($1=="TcpOutSegs")o=d; else r=d} END{printf "out=%d retrans=%d rate=%.3f%%\n",o,r,100*r/o}'

Under 0.1 % is normal internet weather; 0.1–1 % is worth investigating with mtr to find the hop that correlates with the loss; above 1 % is a fault or a saturated link, and no congestion control will hide it. Cross-check TcpExtTCPTimeouts in the same window: timeouts without matching retransmits usually mean the loss is on the return path, which a drop counter on your own interface will never show.

What the numbers look like when fixed

On a 200 ms path with 0.1 % loss, a realistic progression from a stock Ubuntu 24.04 guest looks like this — measured with the same command each time, single stream, 30-second runs, three runs per row:

ConfigurationSingle streamp99 RTT idle → loadedRetrans rate
Stock: cubic, 6 MB max window2.5–3.5 MB/s20 → 380 ms~0.4 %
+ 16 MB window ceiling3.0–4.0 MB/sunchanged~0.4 %
+ fq_codel egress3.0–4.0 MB/s20 → 45 ms~0.4 %
+ BBR8–10 MB/s20 → 45 ms~0.4 %

The important reading: buffers alone changed nothing on this lossy path, fq_codel fixed only the latency, and BBR fixed only the throughput. Three separate problems, three separate fixes. Anyone who tells you “just enable BBR” is measuring one column of this table.

Persist the changes and keep the evidence

Put every sysctl into /etc/sysctl.d/ and the qdisc into a systemd unit that runs after network-online.target, because tc commands do not survive a reboot or a link flap. Keep the three JSON outputs from your final run next to the config so that six months later, when throughput changes, you can tell whether the path changed or your configuration did — that comparison is the only way to distinguish a provider-side change from a regression you introduced.

Leave a Reply