Tuning TCP and NIC Offloads for VPS Throughput: SO_SNDBUF, TSO/GSO, and RTT-Based Buffer Sizing

Network tuning advice for VPS hosts is usually a list of sysctl values copied from a 2013 bare-metal guide. Most of it is wrong in two specific ways: it ignores BDP math, and it ignores virtualized NIC offload behaviour. This article derives the buffer sizes your link actually needs, then shows how to verify that the guest and the host agree about segmentation offloads — the single most common cause of a network path that benchmarks at 10 Gbps but delivers 200 Mbps in production.

Start From the Bandwidth-Delay Product, Not From a Blog Post

Socket buffers exist to keep the pipe full while waiting for an ACK. The amount of data in flight at any moment is BDP = bandwidth × RTT. For a 1 Gbps link with 40 ms RTT:

# 1 Gbit/s = 125 MB/s; RTT 40 ms
# BDP = 125 * 0.040 = 5 MB
# one socket needs ~5 MB in flight -> rmem/wmem of ~6 MB to be safe

Now compute it for the VPS you actually rent and set the values accordingly:

#!/bin/bash
# bdp.sh - print required socket buffers for a link
BW_MBPS=$1   # link speed in Mbit/s
RTT_MS=$2    # measured RTT in ms
python3 - "$BW_MBPS" "$RTT_MS" <<'PY'
import sys
bw = float(sys.argv[1]) / 8 * 1_000_000   # bytes/s
rtt = float(sys.argv[2]) / 1000           # seconds
bdp = bw * rtt
print(f"BDP          = {bdp/1e6:.2f} MB")
print(f"tcp_rmem max = {int(bdp*1.5)}")
print(f"tcp_wmem max = {int(bdp*1.5)}")
PY
./bdp.sh 1000 40

Apply the result and let autotuning work from a correct ceiling rather than an arbitrary one:

cat >> /etc/sysctl.d/99-net-bdp.conf <<'EOF'
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 131072 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_mtu_probing = 2
EOF
sysctl --system

The Real Killer: Offload State Mismatch Across virtio

When a guest sends a 64 KB TCP segment, it depends on the virtual NIC to hand the kernel a segment large enough for the hardware (or host) to split. If the guest advertises TSO/GSO/GRO but the vhost-net backend has them disabled — or the traffic is bridged through a device that cannot handle large segments — the host performs software segmentation of every packet. Throughput collapses and per-packet CPU rises. Check both ends:

# inside the guest
ethtool -k eth0 | egrep 'tso|gso|gro|gso|tx-checksum'
ethtool -S eth0 | egrep 'tx_queue_stop|tx_timeout|seg'

# confirm the virtual device type
ethtool -i eth0
# virtio_net is good; e1000 / rtl8139 emulated NICs are the problem

# verify MSS clamping if traffic is tunnelled
ip -d link show eth0
iptables -t mangle -L -n -v | grep -i mss

If you are on an emulated NIC rather than virtio-net, no amount of sysctl tuning will get you past roughly 1 Gbps of CPU-bound throughput. That is a provider decision: ask about it before you buy, and factor it into how the plans on virtualserversvps.com handle network virtualization.

Measure the Path Before You Tune It

Use iperf3 for raw capacity, but measure it in the right direction (many VPS links are asymmetric), then use ss to see the actual congestion window and RTT the kernel is working with:

# server side
iperf3 -s

# client: 4 parallel streams, 60s, report per-stream
iperf3 -c target.example.net -P 4 -t 60 -O 5 --json | jq '.end.sum_sent.bits_per_second/1e6'

# watch cwnd, rtt and retransmits for a live connection
ss -tin state established '( dport = :443 )' | head -40
# look for: rtt, rttvar, cwnd, retrans, ssthresh

# count retransmissions system-wide
nstat -az TcpRetransSegs TcpExtTCPLostRetransmit

A healthy long-lived connection on a 1 Gbps link with 40 ms RTT should hold cwnd in the 200–400 segment range. If cwnd stays under 20 and retrans climbs, you have loss, not a buffer problem — go find it with mtr and packet captures rather than raising rmem_max.

Symptom	Likely cause	Wrong fix	Right fix
Throughput plateaus far below link speed, low CPU	Socket buffer ceiling too low for BDP	More vCPUs	Raise tcp_rmem/tcp_wmem max
Throughput plateaus, 100% softirq on one core	Offload mismatch, software segmentation	Raise buffers	Enable TSO/GSO/GRO consistently, use virtio-net
Latency spikes under load, buffers full	Bufferbloat in the virtual switch	Bigger buffers	Fair queueing, smaller queues, BBR
Slow single connection, fast parallel	Window-limited single flow	More parallel flows	Fix BDP sizing, check cwnd

Congestion Control and Queue Discipline Together

BBR is a large win on lossy or bufferbloated paths but does nothing for an idle link; fq_codel fixes bufferbloat but relies on the sender behaving. Use both, and verify:

sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq_codel
sysctl -w net.ipv4.tcp_notsent_lowat=131072

sysctl net.ipv4.tcp_available_congestion_control
tc qdisc show dev eth0
ss -tin | grep -o 'bbr' | head -3

tcp_notsent_lowat deserves special mention for request/response workloads: it stops the kernel from queuing megabytes of unacknowledged data that will never help, which measurably reduces TTFB under load without any downside on throughput. If your bottleneck turns out to be application-side rather than kernel-side, the hosting comparisons on virtualserversvps.com cover when moving up a plan beats tuning one.

Verification Loop

• Compute the BDP for your real RTT before setting any buffer value.
• Confirm virtio-net and matching offload flags on both ends of the vhost path.
• Measure with iperf3 -P 4 --json and inspect ss -tin, not just final throughput.
• Add fq_codel plus BBR, then re-measure p99 latency under saturation, not idle throughput.
• Set tcp_notsent_lowat for latency-sensitive APIs.

A tuned VPS network is one where every buffer value can be traced to an arithmetic derivation, and every offload flag has been verified on both sides of the hypervisor boundary.