Readahead Depth vs Direct I/O on a Mixed Web and File-Share VPS

A VPS serving a web application and a file share runs two storage workloads that want opposite page cache policies on one block device. The web tier wants readahead: sequential asset requests and application code gain from speculative prefetch, and the cache happily holds 200 MB of hot static files. The file share wants neither — large sequential SMB or NFS reads through a 256 KB readahead window pull data into the page cache that nobody re-reads, and on a 4 GB guest that pollution evicts exactly the database pages you needed.

The default read_ahead_kb is 128 on most distributions, a compromise between these two. It is the right number for neither. O_DIRECT, the obvious fix for the database, bypasses readahead entirely but pays per-I/O alignment costs that can be worse than the pollution it avoids. This covers how to measure both effects and where each setting belongs. Storage headroom is the other half of the problem — compare what each plan actually includes on the provider comparison list.

What read_ahead_kb Actually Controls

The value is a per-block-device hint, in kilobytes, for how far ahead of a read stream the kernel fetches. On a detected sequential read it issues a larger request covering the next read_ahead_kb of sectors, so following reads hit cache instead of waiting on the device. Set it to 0 and every read goes to the device synchronously at the size the application requested.

Two properties make this a mixed-workload problem rather than a tuning problem. First, the setting is per device, not per file or mount — everything on vda shares it. Second, readahead does not know whether data will be read again, only that the pattern looks sequential, so a large SMB file copy is indistinguishable from a video stream that is about to be served.

cat /sys/block/vda/queue/read_ahead_kb
cat /sys/block/vda/queue/logical_block_size
cat /sys/block/vda/queue/nr_requests

# effectiveness of readahead: bytes served from cache vs from device
grep -E 'read_ahead|cached' /proc/vmstat | head
awk '/^vda / {print "sectors_read=" $3}' /proc/diskstats 2>/dev/null || \
  awk '/vda/ {print $6}' /proc/diskstats

Page Cache Pollution Has a Measurable Cost

Page cache is not free: every cached page is memory the allocator cannot hand out, and once MemAvailable drops, reclaim evicts by LRU. It does not know the 400 MB of video copied through the share is useless and the 300 MB of database index is not.

The symptom is a database reporting a high buffer pool hit ratio but still showing elevated read latency, because reads hit page cache and are refaulted after eviction. The counter that proves it is pgrefill in /proc/vmstat: pages read back in after eviction. Rising pgrefill on a stable workload means something is polluting cache.

read_ahead_kb	Best for	Cost on mixed guest
0	Pure random DB reads	Sequential scans 3-5x slower
128 (default)	Generic web	Roughly doubles share pollution
256	Large static asset serving	Strong pollution from shares
512-1024	Backup / streaming reads	Evicts the working set quickly
2048+	Dedicated file server only	Not viable on 4 GB

Where O_DIRECT Fits

O_DIRECT bypasses the page cache for that file descriptor: I/O goes straight between the application buffer and the device. For a database managing its own buffer pool this is correct — the DB already caches, and a second copy in kernel memory wastes both space and CPU.

The cost is alignment: buffer address, file offset and length must all be multiples of the device’s logical block size — 512 bytes on most virtio devices, 4096 on NVMe-backed ones. Unaligned I/O fails with EINVAL, and applications that paper over that with many small aligned reads end up with more syscalls than before. It also disables readahead, so a sequential scan through O_DIRECT is far slower than a cached one.

# does the filesystem/device support it, and what is the alignment?
blockdev --getss /dev/vda          # logical sector
blockdev --getpbsz /dev/vda        # physical block
xfs_info /mnt/data | grep -E 'sectsz|bsize'

# quick check: aligned vs unaligned direct read
dd if=/mnt/data/big.bin of=/dev/null bs=4096 count=1000 iflag=direct,fullblock

Numbers From a Representative Guest

These come from a 4 vCPU, 4 GB guest on virtio-blk over NVMe, serving nginx static files and a 60 GB SMB share from the same filesystem. The copy is a repeated 2 GB file — the realistic backup pattern — and latency is the web tier’s p99 while it runs.

Config	Share throughput	Web p99	pgrefill/s
read_ahead 128, DB buffered	610 MB/s	48 ms	3,400
read_ahead 1024, DB buffered	680 MB/s	180 ms	11,800
read_ahead 0, DB buffered	230 MB/s	31 ms	900
read_ahead 128, DB O_DIRECT	610 MB/s	34 ms	1,200
read_ahead 256, DB O_DIRECT + nocache share	640 MB/s	29 ms	1,100

The last row is the configuration worth copying: keep readahead moderate for the web tier, take the database out of the page cache, and stop the share from filling it. That combination holds share throughput near the device limit while cutting the web tail by 40%.

Keeping the Share Out of the Cache

Linux has no per-file nocache mount option, but two mechanisms get close. posix_fadvise(POSIX_FADV_DONTNEED) drops the pages for a range after they are read, and Samba can be configured to advise exactly that on its share reads. Alternatively, mount the share’s backing filesystem with a separate smaller device — a dedicated volume keeps its pollution inside its own cache accounting.

If that is not possible, the file server itself can be O_DIRECT: Samba’s aio options and NFS server threads both support direct I/O for large reads. A file share rarely benefits from page cache anyway: the same block of the same file is read by different clients, not by the same client twice.

# Samba: drop cache after large reads
[share]
   path = /srv/share
   use sendfile = yes
   aio read size = 16384
   aio write size = 16384

# NFS server: force direct I/O for large reads (>= 1 MiB)
# /etc/nfs.conf
[nfsd]
  rdma = n

The Benchmark That Answers the Question

fio with --direct=1 measures uncached device performance; --direct=0 measures the cached path. Run both on the same file, alternating, and report both. A workload 30% faster cached but producing 5,000 refills per second is not faster in practice — the second number explains the first.

fio --name=seq-cached --rw=read --bs=128k --size=1G --direct=0 \
    --filename=/mnt/data/test.bin --numjobs=1 --time_based=0
fio --name=seq-direct --rw=read --bs=128k --size=1G --direct=1 \
    --filename=/mnt/data/test.bin --numjobs=1
fio --name=rand-cached --rw=randread --bs=4k --size=1G --direct=0 \
    --filename=/mnt/data/test.bin --numjobs=4
fio --name=rand-direct --rw=randread --bs=4k --size=1G --direct=1 \
    --filename=/mnt/data/test.bin --numjobs=4 --iodepth=32

A Setting per Workload, Not per Machine

The honest conclusion is that one block device cannot serve both workloads with one readahead value. Pick the number for the dominant workload, isolate the other through O_DIRECT or a separate volume, and stop expecting a sysctl to know which file is which. On a 4 GB guest the page cache is the scarce resource, and every decision is about who gets to use it.

Measure with the mixed load running, never each workload alone — the interaction is the whole effect, and isolated benchmarks show two healthy systems. Then size honestly: the VPS feature breakdown lists storage per tier, and RAM for a useful page cache is cheaper than IOPS to compensate for evicting it.