Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files

A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from 900 MB/s to 180 MB/s, r_await on the data device climbs from 0.4 ms to 22 ms, and ss -ti shows the congestion window pinned at around 380 KB on every connection at once. Nothing is broken. The block layer’s read-ahead and TCP’s slow-start are fighting, and the disk lost.

The mechanism is a queue-depth collision. read_ahead_kb makes the kernel issue speculative reads ahead of each sequential stream; with 200 concurrent transfers each demanding 128 KB of look-ahead, the device queue fills with reads nobody asked for yet. Real reads wait, the application drains its socket slowly, the sender’s cwnd stops growing, and every connection sits in congestion avoidance at a fraction of link capacity.

Both tunables are individually correct and individually documented. Together they close a feedback loop where disk latency throttles the stream that would have raised throughput. Instances with a known disk queue and enough memory make the loop easy to observe; the fix is an ordering problem, not a magic sysctl.

What read-ahead actually costs you

read_ahead_kb defaults to 128 KB on most distributions, 32 blocks of speculative read per stream at a 4 KB block size. On one stream that is free throughput. On 200 streams it is 25 MB in flight, and NVMe queues are usually 1024 entries deep at 4–8 KB each. You have filled the queue before the application has read a byte.

Check the value and the resulting concurrency: blockdev --getra /dev/nvme0n1 returns sectors (256 = 128 KB), and iostat -x 1 shows aqu-sz. If aqu-sz exceeds roughly 4× your stream count, read-ahead is inflating the queue. The mixed-workload tell is r_await rising while r/s stays flat or falls.

What cwnd needs from you

TCP exits slow-start when cwnd reaches sndbuf or the congestion threshold, whichever comes first, and cwnd grows only as fast as ACKs arrive. Any receiver-side stall stretches that curve in time. With tcp_rmem capped at 208 KB on some kernels, a stalled reader holds cwnd near the initial window for seconds: 380 KB / 0.04 s RTT is about 9.5 MB/s per connection.

ss -ti separates the two causes cleanly. cwnd below ssthresh with a high rtt and bytes_acked crawling means the receiver is not draining. cwnd at ssthresh with retransmits climbing means actual loss. A persistently full send-q on the server beside a stalled ACK stream is the receiver’s fault, not the path’s.

The measurement pair that identifies the culprit

Run a single controlled test: set read-ahead to 16 KB, rerun the batch, and watch whether r_await and cwnd both move. If they do, the interaction is confirmed and you can tune from either end. If cwnd stays pinned while r_await is unchanged, the bottleneck is elsewhere — path MTU, a middlebox, sender pacing — and no local tunable will help.

iostat -x 1 30 | grep -E 'Device|nvme0n1'
# rising await + flat r/s = queue inflation

ss -tim dport = :443 | egrep 'cwnd|send-q'
# cwnd plateau with low retransmits = receiver-side stall

blockdev --setra 32 /dev/nvme0n1   # 16 KB, test window only

Tuning order: disk first, socket second, never both

Set the disk layer first because it is upstream in the causal chain and its effect is measurable in isolation with fio. Reduce read-ahead on devices serving many concurrent streams, keep it at 128 KB or higher for sequential analytics scans, and verify aqu-sz stays under 4× concurrency. Only then touch net.core.rmem_max and tcp_rmem, and only if ss -ti shows cwnd capped by buffer size.

Changing both at once makes the result uninterpretable and hides a common failure: enlarging socket buffers while read-ahead stays at 512 KB lets each connection queue more data onto a device still thrashing. The apparent gain comes from bigger in-flight windows masking latency, and p99 gets worse as warm-up time lengthens.

Thresholds that separate these cases

One number worth holding in mind: at 128 KB and 200 streams the device sees 25 MB of speculative demand per round. At 8 KB per queue entry that is roughly 3,200 entries, well past a typical 1024-deep queue. Dropping to 16 KB brings it to 400 entries, leaving room for the real reads — usually enough for cwnd to recover unaided.

Observation	Read as	First action
aqu-sz > 4× streams	read-ahead inflation	lower read_ahead_kb
r_await > 5 ms on NVMe	queue contention	lower read-ahead, check neighbours
cwnd < 100 KB steady	receiver stall	find the draining process
cwnd at sndbuf/2	buffer-limited	raise tcp_rmem after disk
retrans > 0.5% of segs	actual loss	not a tunable problem
r/s falling as r_await rises	queue too deep	cut concurrency or read-ahead

Why large files specifically expose this

Small-file traffic spreads across short streams that exit slow-start and close before the queue matters; read-ahead per stream is bounded by file size. Large files keep one sequential stream alive for minutes, so look-ahead stays maximal for the entire transfer and the queue never drains. That is why a static site serves fine and a media backfill stalls exactly when concurrent large transfers cross your queue-depth headroom.

Size it in advance: read_ahead_kb × streams ÷ 8 KB should stay under 60% of queue depth. For 200 streams on a 1024-entry queue that puts the limit at 24 KB, well under the 128 KB default. A udev rule setting per-device read-ahead keeps it stable across reboots instead of relying on a one-off blockdev call.

Verifying the fix without fooling yourself

Re-run the exact batch with the same file sizes and concurrency and compare three numbers: mean throughput, r_await p99, and mean cwnd at the 60-second mark. All three should move together. A throughput gain with unchanged cwnd suggests page cache, not TCP; a cwnd gain with unchanged r_await means you fixed the socket layer instead of the queue.

The lesson generalises: two knobs that both govern buffering will interact, and the interaction appears only at concurrency. Tune the layer closest to the constrained resource, verify with the metric that layer owns, then move outward. On a VPS whose queue depth and memory are known you can compute the safe read-ahead before you ever see the stall, which is the point of renting hardware whose numbers you have.