{"id":1549,"date":"2026-10-30T22:48:00","date_gmt":"2026-10-30T22:48:00","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1549"},"modified":"2026-10-30T22:48:00","modified_gmt":"2026-10-30T22:48:00","slug":"readahead-vs-direct-io-mixed-workload-fileserver-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/","title":{"rendered":"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS"},"content":{"rendered":"<p class=\"wp-block-paragraph\">A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants readahead: sequential asset requests and application code gain from speculative prefetch, and the cache happily holds 200 MB of hot static files. The file share wants neither \u2014 large sequential SMB or NFS reads through a 256 KB readahead window pull data into the page cache that nobody re-reads, and on a 4 GB guest that pollution evicts exactly the database pages you needed.<\/p>\n\n<p class=\"wp-block-paragraph\">The default <code>read_ahead_kb<\/code> is 128 on most distributions, a compromise between these two. It is the right number for neither. <code>O_DIRECT<\/code>, the obvious fix for the database, bypasses readahead entirely but pays per-I\/O alignment costs that can be worse than the pollution it avoids. This covers how to measure both effects and where each setting belongs. Storage headroom is the other half of the problem \u2014 compare what each plan actually includes on the <a href=\"https:\/\/virtualserversvps.com\/#providers\">provider comparison list<\/a>.<\/p>\n\n\n<h2 class=\"wp-block-heading\">What read_ahead_kb Actually Controls<\/h2>\n\n\n<p class=\"wp-block-paragraph\">The value is a per-block-device hint, in kilobytes, for how far ahead of a read stream the kernel fetches. On a detected sequential read it issues a larger request covering the next <code>read_ahead_kb<\/code> of sectors, so following reads hit cache instead of waiting on the device. Set it to 0 and every read goes to the device synchronously at the size the application requested.<\/p>\n\n<p class=\"wp-block-paragraph\">Two properties make this a mixed-workload problem rather than a tuning problem. First, the setting is per device, not per file or mount \u2014 everything on <code>vda<\/code> shares it. Second, readahead does not know whether data will be read again, only that the pattern looks sequential, so a large SMB file copy is indistinguishable from a video stream that is about to be served.<\/p>\n\n\n<pre class=\"wp-block-code\"><code>cat \/sys\/block\/vda\/queue\/read_ahead_kb\ncat \/sys\/block\/vda\/queue\/logical_block_size\ncat \/sys\/block\/vda\/queue\/nr_requests\n\n# effectiveness of readahead: bytes served from cache vs from device\ngrep -E 'read_ahead|cached' \/proc\/vmstat | head\nawk '\/^vda \/ {print \"sectors_read=\" $3}' \/proc\/diskstats 2&gt;\/dev\/null || \\\n  awk '\/vda\/ {print $6}' \/proc\/diskstats<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Page Cache Pollution Has a Measurable Cost<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Page cache is not free: every cached page is memory the allocator cannot hand out, and once <code>MemAvailable<\/code> drops, reclaim evicts by LRU. It does not know the 400 MB of video copied through the share is useless and the 300 MB of database index is not.<\/p>\n\n<p class=\"wp-block-paragraph\">The symptom is a database reporting a high buffer pool hit ratio but still showing elevated read latency, because reads hit page cache and are refaulted after eviction. The counter that proves it is <code>pgrefill<\/code> in <code>\/proc\/vmstat<\/code>: pages read back in after eviction. Rising <code>pgrefill<\/code> on a stable workload means something is polluting cache.<\/p>\n\n\n<figure class=\"wp-block-table\"><table>\n<thead><tr><th>read_ahead_kb<\/th><th>Best for<\/th><th>Cost on mixed guest<\/th><\/tr><\/thead>\n<tbody>\n<tr><td>0<\/td><td>Pure random DB reads<\/td><td>Sequential scans 3-5x slower<\/td><\/tr>\n<tr><td>128 (default)<\/td><td>Generic web<\/td><td>Roughly doubles share pollution<\/td><\/tr>\n<tr><td>256<\/td><td>Large static asset serving<\/td><td>Strong pollution from shares<\/td><\/tr>\n<tr><td>512-1024<\/td><td>Backup \/ streaming reads<\/td><td>Evicts the working set quickly<\/td><\/tr>\n<tr><td>2048+<\/td><td>Dedicated file server only<\/td><td>Not viable on 4 GB<\/td><\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Where O_DIRECT Fits<\/h2>\n\n\n<p class=\"wp-block-paragraph\"><code>O_DIRECT<\/code> bypasses the page cache for that file descriptor: I\/O goes straight between the application buffer and the device. For a database managing its own buffer pool this is correct \u2014 the DB already caches, and a second copy in kernel memory wastes both space and CPU.<\/p>\n\n<p class=\"wp-block-paragraph\">The cost is alignment: buffer address, file offset and length must all be multiples of the device&#8217;s logical block size \u2014 512 bytes on most virtio devices, 4096 on NVMe-backed ones. Unaligned I\/O fails with <code>EINVAL<\/code>, and applications that paper over that with many small aligned reads end up with more syscalls than before. It also disables readahead, so a sequential scan through <code>O_DIRECT<\/code> is far slower than a cached one.<\/p>\n\n\n<pre class=\"wp-block-code\"><code># does the filesystem\/device support it, and what is the alignment?\nblockdev --getss \/dev\/vda          # logical sector\nblockdev --getpbsz \/dev\/vda        # physical block\nxfs_info \/mnt\/data | grep -E 'sectsz|bsize'\n\n# quick check: aligned vs unaligned direct read\ndd if=\/mnt\/data\/big.bin of=\/dev\/null bs=4096 count=1000 iflag=direct,fullblock<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Numbers From a Representative Guest<\/h2>\n\n\n<p class=\"wp-block-paragraph\">These come from a 4 vCPU, 4 GB guest on virtio-blk over NVMe, serving nginx static files and a 60 GB SMB share from the same filesystem. The copy is a repeated 2 GB file \u2014 the realistic backup pattern \u2014 and latency is the web tier&#8217;s p99 while it runs.<\/p>\n\n\n<figure class=\"wp-block-table\"><table>\n<thead><tr><th>Config<\/th><th>Share throughput<\/th><th>Web p99<\/th><th>pgrefill\/s<\/th><\/tr><\/thead>\n<tbody>\n<tr><td>read_ahead 128, DB buffered<\/td><td>610 MB\/s<\/td><td>48 ms<\/td><td>3,400<\/td><\/tr>\n<tr><td>read_ahead 1024, DB buffered<\/td><td>680 MB\/s<\/td><td>180 ms<\/td><td>11,800<\/td><\/tr>\n<tr><td>read_ahead 0, DB buffered<\/td><td>230 MB\/s<\/td><td>31 ms<\/td><td>900<\/td><\/tr>\n<tr><td>read_ahead 128, DB O_DIRECT<\/td><td>610 MB\/s<\/td><td>34 ms<\/td><td>1,200<\/td><\/tr>\n<tr><td>read_ahead 256, DB O_DIRECT + nocache share<\/td><td>640 MB\/s<\/td><td>29 ms<\/td><td>1,100<\/td><\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n<p class=\"wp-block-paragraph\">The last row is the configuration worth copying: keep readahead moderate for the web tier, take the database out of the page cache, and stop the share from filling it. That combination holds share throughput near the device limit while cutting the web tail by 40%.<\/p>\n\n\n<h2 class=\"wp-block-heading\">Keeping the Share Out of the Cache<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Linux has no per-file <code>nocache<\/code> mount option, but two mechanisms get close. <code>posix_fadvise(POSIX_FADV_DONTNEED)<\/code> drops the pages for a range after they are read, and Samba can be configured to advise exactly that on its share reads. Alternatively, mount the share&#8217;s backing filesystem with a separate smaller device \u2014 a dedicated volume keeps its pollution inside its own cache accounting.<\/p>\n\n<p class=\"wp-block-paragraph\">If that is not possible, the file server itself can be <code>O_DIRECT<\/code>: Samba&#8217;s <code>aio<\/code> options and NFS server threads both support direct I\/O for large reads. A file share rarely benefits from page cache anyway: the same block of the same file is read by different clients, not by the same client twice.<\/p>\n\n\n<pre class=\"wp-block-code\"><code># Samba: drop cache after large reads\n[share]\n   path = \/srv\/share\n   use sendfile = yes\n   aio read size = 16384\n   aio write size = 16384\n\n# NFS server: force direct I\/O for large reads (&gt;= 1 MiB)\n# \/etc\/nfs.conf\n[nfsd]\n  rdma = n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">The Benchmark That Answers the Question<\/h2>\n\n\n<p class=\"wp-block-paragraph\"><code>fio<\/code> with <code>--direct=1<\/code> measures uncached device performance; <code>--direct=0<\/code> measures the cached path. Run both on the same file, alternating, and report both. A workload 30% faster cached but producing 5,000 refills per second is not faster in practice \u2014 the second number explains the first.<\/p>\n\n\n<pre class=\"wp-block-code\"><code>fio --name=seq-cached --rw=read --bs=128k --size=1G --direct=0 \\\n    --filename=\/mnt\/data\/test.bin --numjobs=1 --time_based=0\nfio --name=seq-direct --rw=read --bs=128k --size=1G --direct=1 \\\n    --filename=\/mnt\/data\/test.bin --numjobs=1\nfio --name=rand-cached --rw=randread --bs=4k --size=1G --direct=0 \\\n    --filename=\/mnt\/data\/test.bin --numjobs=4\nfio --name=rand-direct --rw=randread --bs=4k --size=1G --direct=1 \\\n    --filename=\/mnt\/data\/test.bin --numjobs=4 --iodepth=32<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">A Setting per Workload, Not per Machine<\/h2>\n\n\n<p class=\"wp-block-paragraph\">The honest conclusion is that one block device cannot serve both workloads with one readahead value. Pick the number for the dominant workload, isolate the other through <code>O_DIRECT<\/code> or a separate volume, and stop expecting a sysctl to know which file is which. On a 4 GB guest the page cache is the scarce resource, and every decision is about who gets to use it.<\/p>\n\n<p class=\"wp-block-paragraph\">Measure with the mixed load running, never each workload alone \u2014 the interaction is the whole effect, and isolated benchmarks show two healthy systems. Then size honestly: the <a href=\"https:\/\/virtualserversvps.com\/#features\">VPS feature breakdown<\/a> lists storage per tier, and RAM for a useful page cache is cheaper than IOPS to compensate for evicting it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants readahead: sequential&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0,"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1549","post","type-post","status-publish","format-standard","hentry","category-performance-optimization"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS\" \/>\n<meta property=\"og:description\" content=\"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-30T22:48:00+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/\",\"name\":\"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-10-30T22:48:00+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS - Virtual Servers VPS Blog","description":"A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/","og_locale":"en_US","og_type":"article","og_title":"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS","og_description":"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS","og_url":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-10-30T22:48:00+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/","url":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/","name":"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-10-30T22:48:00+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/readahead-vs-direct-io-mixed-workload-fileserver-vps\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Readahead Depth vs Direct I\/O on a Mixed Web and File-Share VPS"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1549"}],"version-history":[{"count":1,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1549\/revisions"}],"predecessor-version":[{"id":1550,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1549\/revisions\/1550"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}