BBR + fq vs CAKE + fq_codel: Picking a Queue Discipline on a Single-vCPU VPS

Everyone has already enabled BBR. The question on a single-vCPU box is not which congestion control to use, it is what you pair it with — BBR plus fq, or the fq_codel queueing that CAKE builds on. Both work; they fail differently under saturation, and the CPU cost is not the same when you only have one core. This is a measurement methodology you can run on a $10 instance in about forty minutes, with the specific counters to read. Whether your provider’s kernel exposes the needed qdiscs is a separate question worth checking in the VPS provider list supporting modern kernels.

Why the Pairing Matters More Than BBR

BBR is a congestion control algorithm: it decides send rate. fq and CAKE are queue disciplines: they decide what happens to packets waiting to leave the interface. BBR estimates bottleneck bandwidth and round-trip time and paces packets to match. That pacing assumes a queue discipline that spreads packets out rather than dumping bursts — which is why the canonical pairing is BBR + fq. CAKE does the same spreading plus per-host fairness and optional shaping, but it computes a shaper and a hash per flow, and on one vCPU that computation competes with your application.

sysctl -n net.ipv4.tcp_congestion_control
sysctl -n net.core.default_qdisc
tc qdisc show dev eth0

Expected on a stock Ubuntu 24.04 kernel: cubic, fq_codel, and qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms. If you see noqueue, the interface is a virtual NIC that hands packets straight to the hypervisor and qdisc changes will have limited effect — check that before benchmarking anything.

The Two Stacks, Precisely

# Stack A: BBR + fq
sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr
tc qdisc replace dev eth0 root fq maxrate 900mbit

# Stack B: CAKE + fq_codel leaves (CAKE's internal default)
sysctl -w net.core.default_qdisc=fq_codel
sysctl -w net.ipv4.tcp_congestion_control=bbr
tc qdisc replace dev eth0 root cake bandwidth 900mbit diffserv4

Two details people skip. First, maxrate on fq is what makes BBR pace properly; without it fq spreads packets but doesn’t shape, and you lose the pacing benefit at rates above the link. Second, CAKE’s bandwidth must be set below the real link rate — 900 Mbit on a 1 Gbit port — or the shaper never engages and CAKE behaves like a more expensive fq_codel. If you set the bandwidth above the physical rate, you are measuring nothing.

What to Measure, and With What

CPU cost, latency under load, retransmits, and throughput — in that order of importance on a single vCPU. A stack that gains 5% throughput and costs 20% CPU is a net loss when that CPU runs your web server.

# CPU cost of the qdisc itself, excluding the app
pidstat -u 1 30 | awk '/%CPU/ || /ksoftirqd|kworker/'

# Per-socket state during the test
ss -tin dst TARGET_IP | grep -E 'bbr|cwnd|rtt|retrans|snd'

# Retransmit and loss counters
nstat -az TcpRetransSegs TcpOutSegs TcpExtTCPLostRetransmit

The ss -tin output is where the useful signal lives. On BBR you will see bbr in the algorithm field along with bw: and mrtt: estimates. rcv_ssthresh shows how much receive window the kernel is willing to open — if it’s pinned low, you’re being limited by the receiver or by an undersized socket buffer, not by the congestion control, and the whole A/B test is measuring the wrong thing.

Test 1: Bufferbloat Under Saturation

Generate a saturating upload from one client while pinging a third host, and look at RTT inflation rather than average throughput.

# Terminal 1: saturate the uplink
iperf3 -c TARGET -t 60 -P 4

# Terminal 2: latency under load, 100 pings at 200ms
ping -i 0.2 -c 300 OTHER_HOST | tail -3

Baseline idle RTT to the third host: 14 ms. Under load with fq_codel and no shaping, expect 60–180 ms and jitter — the classic bufferbloat signature, because the qdisc has no target rate to pace against. With BBR + fq maxrate 900mbit, RTT under load typically lands at 16–28 ms. With CAKE at 900 Mbit, 16–24 ms with noticeably tighter jitter. The difference between A and B here is usually small; the difference between either and leaving the default fq_codel is large. That comparison is the one worth running first.

Test 2: Retransmits and Loss Recovery

nstat -n > /dev/null            # reset counters
iperf3 -c TARGET -t 60 -P 4
nstat TcpRetransSegs TcpOutSegs TcpExtTCPLostRetransmit
awk '/TcpRetransSegs|TcpOutSegs/ {print $2, $3}' /proc/net/netstat

Compute the ratio, not the absolute count: TcpRetransSegs / TcpOutSegs. Under 0.1% is clean. Between 0.1% and 1% suggests a lossy path. Above 1% and BBR is doing its job anyway — it reacts to delivery rate rather than loss — but your users see stalls. If CAKE shows materially lower retransmits at the same throughput, the shaper is smoothing bursts that were overflowing a downstream buffer. If the numbers are identical, the downstream bottleneck is elsewhere and the qdisc choice is not your problem.

Test 3: CPU Cost on One vCPU

Run the same 60-second iperf3 three times per stack and record ksoftirqd plus total system CPU from pidstat. On a single vCPU with one 1 Gbps flow, the realistic numbers are: fq at roughly 3–6% of the core, cake at roughly 12–22% with diffserv4. CAKE’s per-packet hashing, DSCP classification and shaper token accounting cost real cycles, and those cycles come out of the same core running nginx.

# Isolate: one stream, then ten
for p in 1 4 10; do
  iperf3 -c TARGET -t 30 -P $p --json | jq ".end.sum_sent.bits_per_second/1e6"
done

Watch how the CPU gap scales with stream count. fq’s cost is close to flat — more flows just means more hash buckets. CAKE’s rises because per-flow classification continues and the shaper does more work per second at higher packet rates. Past roughly 8,000 pps of sustained traffic on one vCPU, CAKE becomes a visible fraction of your headroom.

When Each Wins

Pitfalls That Invalidate the Test

• Changing both variables at once. Congestion control and qdisc must be varied independently, or you can’t attribute the result.
• Benchmarking over Wi-Fi or a shared laptop uplink. Your client’s own buffer dominates the RTT measurement.
• Forgetting tc qdisc replace vs add. add stacks a second qdisc and you measure the wrong one.
• Not resetting counters between runs — nstat accumulates and the retransmit ratio drifts.
• Testing at 900 Mbit on a 100 Mbit plan. The shaper never engages and every result is noise.

Write the per-run numbers into a file and keep them. Kernel versions change CAKE’s defaults and BBR’s pacing behaviour between releases, and a result from 6.5 is not a result on 6.8. Re-run the three tests when you upgrade the box or move providers — the pair that wins on a 1 Gbps port with 20 ms of headroom is not necessarily the pair that wins on a 200 Mbit port. If you’re starting from a fresh instance, the VPS plans with root-level tc access are the ones where any of this is testable at all.