{"id":1572,"date":"2026-11-01T22:35:56","date_gmt":"2026-11-01T22:35:56","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1572"},"modified":"2026-11-01T22:35:56","modified_gmt":"2026-11-01T22:35:56","slug":"readahead-tcp-cwnd-collapse-slow-start","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/","title":{"rendered":"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from 900 MB\/s to 180 MB\/s, <code>r_await<\/code> on the data device climbs from 0.4 ms to 22 ms, and <code>ss -ti<\/code> shows the congestion window pinned at around 380 KB on every connection at once. Nothing is broken. The block layer&#8217;s read-ahead and TCP&#8217;s slow-start are fighting, and the disk lost.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The mechanism is a queue-depth collision. <code>read_ahead_kb<\/code> makes the kernel issue speculative reads ahead of each sequential stream; with 200 concurrent transfers each demanding 128 KB of look-ahead, the device queue fills with reads nobody asked for yet. Real reads wait, the application drains its socket slowly, the sender&#8217;s cwnd stops growing, and every connection sits in congestion avoidance at a fraction of link capacity.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Both tunables are individually correct and individually documented. Together they close a feedback loop where disk latency throttles the stream that would have raised throughput. <a href=\"https:\/\/virtualserversvps.com\/\">Instances with a known disk queue and enough memory<\/a> make the loop easy to observe; the fix is an ordering problem, not a magic sysctl.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What read-ahead actually costs you<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><code>read_ahead_kb<\/code> defaults to 128 KB on most distributions, 32 blocks of speculative read per stream at a 4 KB block size. On one stream that is free throughput. On 200 streams it is 25 MB in flight, and NVMe queues are usually 1024 entries deep at 4\u20138 KB each. You have filled the queue before the application has read a byte.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Check the value and the resulting concurrency: <code>blockdev --getra \/dev\/nvme0n1<\/code> returns sectors (256 = 128 KB), and <code>iostat -x 1<\/code> shows <code>aqu-sz<\/code>. If <code>aqu-sz<\/code> exceeds roughly 4\u00d7 your stream count, read-ahead is inflating the queue. The mixed-workload tell is <code>r_await<\/code> rising while <code>r\/s<\/code> stays flat or falls.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What cwnd needs from you<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">TCP exits slow-start when cwnd reaches <code>sndbuf<\/code> or the congestion threshold, whichever comes first, and cwnd grows only as fast as ACKs arrive. Any receiver-side stall stretches that curve in time. With <code>tcp_rmem<\/code> capped at 208 KB on some kernels, a stalled reader holds cwnd near the initial window for seconds: 380 KB \/ 0.04 s RTT is about 9.5 MB\/s per connection.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>ss -ti<\/code> separates the two causes cleanly. cwnd below <code>ssthresh<\/code> with a high <code>rtt<\/code> and <code>bytes_acked<\/code> crawling means the receiver is not draining. cwnd at <code>ssthresh<\/code> with retransmits climbing means actual loss. A persistently full <code>send-q<\/code> on the server beside a stalled ACK stream is the receiver&#8217;s fault, not the path&#8217;s.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The measurement pair that identifies the culprit<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Run a single controlled test: set read-ahead to 16 KB, rerun the batch, and watch whether <code>r_await<\/code> and <code>cwnd<\/code> both move. If they do, the interaction is confirmed and you can tune from either end. If cwnd stays pinned while <code>r_await<\/code> is unchanged, the bottleneck is elsewhere \u2014 path MTU, a middlebox, sender pacing \u2014 and no local tunable will help.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>iostat -x 1 30 | grep -E 'Device|nvme0n1'\n# rising await + flat r\/s = queue inflation\n\nss -tim dport = :443 | egrep 'cwnd|send-q'\n# cwnd plateau with low retransmits = receiver-side stall\n\nblockdev --setra 32 \/dev\/nvme0n1   # 16 KB, test window only<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Tuning order: disk first, socket second, never both<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Set the disk layer first because it is upstream in the causal chain and its effect is measurable in isolation with <code>fio<\/code>. Reduce read-ahead on devices serving many concurrent streams, keep it at 128 KB or higher for sequential analytics scans, and verify <code>aqu-sz<\/code> stays under 4\u00d7 concurrency. Only then touch <code>net.core.rmem_max<\/code> and <code>tcp_rmem<\/code>, and only if <code>ss -ti<\/code> shows cwnd capped by buffer size.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Changing both at once makes the result uninterpretable and hides a common failure: enlarging socket buffers while read-ahead stays at 512 KB lets each connection queue more data onto a device still thrashing. The apparent gain comes from bigger in-flight windows masking latency, and p99 gets worse as warm-up time lengthens.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Thresholds that separate these cases<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">One number worth holding in mind: at 128 KB and 200 streams the device sees 25 MB of speculative demand per round. At 8 KB per queue entry that is roughly 3,200 entries, well past a typical 1024-deep queue. Dropping to 16 KB brings it to 400 entries, leaving room for the real reads \u2014 usually enough for cwnd to recover unaided.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Observation<\/th><th>Read as<\/th><th>First action<\/th><\/tr><\/thead><tbody><tr><td><code>aqu-sz<\/code> &gt; 4\u00d7 streams<\/td><td>read-ahead inflation<\/td><td>lower <code>read_ahead_kb<\/code><\/td><\/tr><tr><td><code>r_await<\/code> &gt; 5 ms on NVMe<\/td><td>queue contention<\/td><td>lower read-ahead, check neighbours<\/td><\/tr><tr><td><code>cwnd<\/code> &lt; 100 KB steady<\/td><td>receiver stall<\/td><td>find the draining process<\/td><\/tr><tr><td><code>cwnd<\/code> at <code>sndbuf<\/code>\/2<\/td><td>buffer-limited<\/td><td>raise <code>tcp_rmem<\/code> after disk<\/td><\/tr><tr><td><code>retrans<\/code> &gt; 0.5% of segs<\/td><td>actual loss<\/td><td>not a tunable problem<\/td><\/tr><tr><td><code>r\/s<\/code> falling as <code>r_await<\/code> rises<\/td><td>queue too deep<\/td><td>cut concurrency or read-ahead<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Why large files specifically expose this<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Small-file traffic spreads across short streams that exit slow-start and close before the queue matters; read-ahead per stream is bounded by file size. Large files keep one sequential stream alive for minutes, so look-ahead stays maximal for the entire transfer and the queue never drains. That is why a static site serves fine and a media backfill stalls exactly when concurrent large transfers cross your queue-depth headroom.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Size it in advance: <code>read_ahead_kb \u00d7 streams \u00f7 8 KB<\/code> should stay under 60% of queue depth. For 200 streams on a 1024-entry queue that puts the limit at 24 KB, well under the 128 KB default. A udev rule setting per-device read-ahead keeps it stable across reboots instead of relying on a one-off <code>blockdev<\/code> call.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Verifying the fix without fooling yourself<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Re-run the exact batch with the same file sizes and concurrency and compare three numbers: mean throughput, <code>r_await<\/code> p99, and mean <code>cwnd<\/code> at the 60-second mark. All three should move together. A throughput gain with unchanged cwnd suggests page cache, not TCP; a cwnd gain with unchanged <code>r_await<\/code> means you fixed the socket layer instead of the queue.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The lesson generalises: two knobs that both govern buffering will interact, and the interaction appears only at concurrency. Tune the layer closest to the constrained resource, verify with the metric that layer owns, then move outward. On a <a href=\"https:\/\/virtualserversvps.com\/\">VPS whose queue depth and memory are known<\/a> you can compute the safe read-ahead before you ever see the stall, which is the point of renting hardware whose numbers you have.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from 900 MB\/s&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":1,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-1572","post","type-post","status-publish","format-standard","hentry","category-provider-reviews-comparisons"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files\" \/>\n<meta property=\"og:description\" content=\"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-01T22:35:56+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/\",\"name\":\"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-11-01T22:35:56+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files - Virtual Servers VPS Blog","description":"A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/","og_locale":"en_US","og_type":"article","og_title":"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files","og_description":"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files","og_url":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-11-01T22:35:56+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/","url":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/","name":"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-11-01T22:35:56+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"A server pulling 40 GB of large files for a CDN backfill behaves consistently for the first 90 seconds and then falls off a cliff. Throughput drops from","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/readahead-tcp-cwnd-collapse-slow-start\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Read-Ahead Versus TCP Slow-Start: Two CPMs That Punish Large Files"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1572","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1572"}],"version-history":[{"count":1,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1572\/revisions"}],"predecessor-version":[{"id":1573,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1572\/revisions\/1573"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1572"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1572"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1572"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}