Tuning net.ipv4.tcp_rmem for High-Latency VPS Workloads: Buffer Sizing by Bandwidth-Delay Product

Default receive-buffer autotuning on Linux fails in one specific situation: a high bandwidth-delay product. On a VPS talking to clients 150–250 ms away, the default net.ipv4.tcp_rmem maximum of 6 MB caps the receive window and, with it, single-stream throughput. This article measures the ceiling, shows how to read the relevant counters, and gives buffer values that match real latency instead of the localhost assumptions the defaults were built for.

Read the current values before changing anything

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.ipv4.tcp_moderate_rcvbuf
sysctl net.core.rmem_max net.core.wmem_max
# actual per-socket buffers in use, live
ss -timm | grep -A1 'ESTAB' | head -20
nstat -az TcpExtTCPRcvChecksumError 2>/dev/null; nstat -az | grep -i win

tcp_rmem has three fields: minimum, default, and maximum. Autotuning (controlled by tcp_moderate_rcvbuf) moves a socket between minimum and maximum based on measured throughput, but it cannot exceed the maximum, and it cannot exceed net.core.rmem_max for explicit SO_RCVBUF requests either. The buffer that matters is whichever of the two is smaller.

The bandwidth-delay product sets the floor

To keep a pipe full, the receive window must hold one round-trip’s worth of data. The arithmetic is BDP = bandwidth × RTT. At 1 Gbit/s and 200 ms RTT: 1,000,000,000 bits/s × 0.2 s = 200,000,000 bits = 25 MB. A 6 MB maximum receive buffer is less than a quarter of that, so a single TCP stream can never fill the link no matter how fast the server is.

RTT to client	BDP at 100 Mbit/s	BDP at 1 Gbit/s	BDP at 10 Gbit/s	Default 6 MB window covers
10 ms (same region)	0.125 MB	1.25 MB	12.5 MB	All except 10G
50 ms (cross-country)	0.6 MB	6.25 MB	62.5 MB	Up to ~1 Gbit/s
120 ms (transatlantic)	1.5 MB	15 MB	150 MB	Only ~400 Mbit/s
200 ms (transpacific)	2.5 MB	25 MB	250 MB	Only ~240 Mbit/s

The last column is the practical finding: with default buffers, a transpacific 1 Gbit/s uplink tops out near 240 Mbit/s per stream. Multi-stream transfers (browsers, rsync with parallel streams, HTTP/2) mask this because each stream gets its own window, which is why a speed test can look fine while a single scp crawls. If you are choosing an instance partly on the strength of its network tier, this per-stream ceiling is worth factoring into the VPS platform comparison rather than trusting a headline port speed.

Tuning values and the memory bill

# /etc/sysctl.d/99-tcp-buffers.conf
net.core.rmem_max = 33554432          # 32 MB hard ceiling
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 1048576 33554432   # min  default  max
net.ipv4.tcp_wmem = 4096 1048576 33554432
net.ipv4.tcp_moderate_rcvbuf = 1     # keep autotuning on
net.ipv4.tcp_window_scaling = 1       # required above 64 KB windows
net.ipv4.tcp_slow_start_after_idle = 0
net.core.optmem_max = 65536

Two points about cost. First, buffers are allocated on demand, not at socket creation, so the maxima are ceilings rather than reservations — the kernel grows a socket’s buffer only as autotuning sees fit. Second, the cost is real under load: 500 concurrent connections each claiming a 32 MB window can account for gigabytes, which is why the maximum belongs sized to your RAM, not to the largest number you have seen someone else use. On a 1 GB instance, 8–16 MB maxima are a reasonable compromise; on 4 GB or more, 32 MB is comfortable. How much RAM an instance actually offers, and how much of it the provider reserves for the host, matters here — see how VPS memory allocation works in practice before sizing maxima against a headline figure.

Verify with a real transfer, not a speed test

Apply with sysctl --system, then measure a single stream to a distant host and watch the window grow:

# run the transfer, then sample the window on the live socket
ss -timm dst 203.0.113.10 | grep -E 'rcv_space|rcv_wnd|wscale|rtt'
# server-side counter check
cat /proc/net/snmp | grep -A1 '^Tcp:'
# sanity: confirm the kernel accepted the new maximum
sysctl net.ipv4.tcp_rmem net.core.rmem_max

In ss -timm output, rcv_space is the buffer the kernel has grown for that socket and rtt is the measured round trip — multiply the two by the throughput and you can confirm the window matches the BDP. If rcv_space stops at the old maximum, either the sysctl did not apply or the socket was opened before the change; new connections pick up new sysctls, existing ones do not. If rcv_space grows to 32 MB but throughput is unchanged, the bottleneck is not the receive window and you are chasing the wrong layer — check for retransmits with ss -ti (look at retrans) before tuning buffers further.

Receive and send buffers are not symmetric

Most guides tune tcp_rmem and stop. That is correct for a server whose dominant traffic is outbound — a file host, a game server, a video origin serving uploads — because the receive window is the constraint on how fast the sender may push. For a server whose dominant traffic is inbound transfers (backups pulled down, packages fetched, a scraper), the equivalent limit is tcp_wmem, and the same bandwidth-delay arithmetic applies. Tune both when you do not know which direction dominates, and check the average socket direction in ss -s or nstat -az | grep -i segs_out versus segs_in to find out.

Interaction with congestion control

Buffer sizing and congestion control are separate levers that are often conflated. BBR (congestion control) estimates bandwidth and RTT and is not loss-sensitive, so it can reach high throughput on lossy long-haul links where loss-based CUBIC stalls. A large receive window and a BBR sender together are what actually produce near-BDP throughput. Enabling BBR without widening the buffers leaves the window as the ceiling; widening the buffers without BBR leaves loss sensitivity in place. Confirm what is active before concluding buffers are the problem:

sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_available_congestion_control
# enable BBR only after confirming the module is available
modprobe tcp_bbr && sysctl -w net.ipv4.tcp_congestion_control=bbr
# persistence
echo 'net.ipv4.tcp_congestion_control=bbr' >> /etc/sysctl.d/99-tcp-buffers.conf

Test the change with a real transfer to a remote host, not a localhost benchmark. A loopback transfer never traverses the network stack’s window limits in the same way and will report line-rate no matter how small the buffers are. Measure against a host at realistic RTT, and record the before/after rcv_space value for the same flow so the comparison is anchored to a number rather than a feeling.