VPS Disk I/O Latency: Measuring p99 with fio, blkio cgroups, and I/O Schedulers

Throughput numbers lie. A disk that reports 500 MB/s sequential can still deliver 12 ms p99 write latency under a mixed read/write database workload, and that tail latency is what your application actually feels. This guide is about measuring latency percentiles, not averages, and then fixing the four kernel-level causes that dominate VPS storage behaviour: queue depth, scheduler choice, cgroup throttling, and filesystem journaling.

Step 1: Measure With Latency Reporting, Not Just Bandwidth

Most people run fio with a job description that only prints bandwidth. Force percentile reporting and mixed workloads. Save this as io-latency.fio:

[global]
ioengine=libaio
direct=1
runtime=60
time_based=1
ramp_time=5
group_reporting=1
percentile_list=50:90:99:99.9
size=2G
filename=/mnt/testfile

[4k-randwrite]
bs=4k
rw=randwrite
iodepth=32
numjobs=4

[4k-randread]
bs=4k
rw=randread
iodepth=32
numjobs=4

[70-30-mixed]
bs=16k
rw=randrw
rwmixread=70
iodepth=64
numjobs=2

Run it with fio io-latency.fio --output-format=json > fio.json and read the write.clat_ns.percentile block. On a healthy KVM VPS with NVMe-backed storage you should see approximately:

Workload	p50	p99	p99.9
4k randwrite, iodepth 32	80–150 µs	<600 µs	<2 ms
4k randread, iodepth 32	60–110 µs	<500 µs	<1.5 ms
16k 70/30 mixed	150–300 µs	<1.2 ms	<4 ms
Same test, throttled VPS	400 µs	8–25 ms	40 ms+

A p99 that is 20x the p50 is the signature of throttling, not slow media. If p50 looks fine and p99 explodes, the problem is upstream of the drive.

Step 2: Find the Cause

Queue depth: the iodepth trap

NVMe sustains high IOPS only when it has enough outstanding requests. A single-threaded, iodepth=1 test measures round-trip latency, not device capability, and makes the platform look far worse than it is. Conversely, pushing iodepth=256 onto a virtio-blk device with a 128-tag queue means requests queue in the guest kernel, inflating clat without ever reaching the host. Check the real queue limits before blaming the provider:

cat /sys/block/vda/queue/nr_requests
cat /sys/block/vda/queue/scheduler
cat /sys/block/vda/queue/rotational
lsblk -d -o NAME,ROTA,SCHED,DISC-GRAN
# virtio-blk exposes limited tags; virtio-scsi + io_uring scales better
sudo lshw -class disk | grep -A2 "logical name"

Scheduler: mq-deadline vs none vs kyber

Since kernel 5.x, NVMe devices default to none and SATA SSDs often to mq-deadline. On a VPS the virtual block device may report itself as rotational, which silently applies scheduler overhead. Test all three and compare p99, not average throughput:

for s in none mq-deadline kyber bfq; do
  echo $s | sudo tee /sys/block/vda/queue/scheduler
  fio --name=w --rw=randwrite --bs=4k --iodepth=32 --direct=1 \
      --size=1G --runtime=30 --time_based --filename=/mnt/testfile \
      --percentile_list=50:99:99.9 2>/dev/null | grep -E "clat|iops"
done

Make it persistent in a udev rule rather than echoing at boot: ACTION=="add|change", KERNEL=="vda", ATTR{queue/scheduler}="none" in /etc/udev/rules.d/60-io-scheduler.rules. A related deep dive on the same trade-offs for kernel parameters is worth reading alongside this: the performance tuning references at virtualserversvps.com.

cgroup v2 io.max throttling

This is the most common hidden cause of p99 spikes on a VPS. Providers enforce disk quotas through io.max in the host cgroup, and when your burst budget is consumed, requests are delayed rather than rejected — producing multi-millisecond stalls with no obvious error. You cannot always read the host cgroup, but you can read pressure stall information, which correlates almost perfectly with provider throttling:

cat /proc/pressure/io
# some avg10=0.22 avg60=0.18 avg300=0.12 total=8421331
# full avg10=0.09 avg60=0.07 avg300=0.05 total=3119088

# per-cgroup pressure, if delegated
cat /sys/fs/cgroup/$(systemd-cgls --no-pager >/dev/null; echo system.slice)/io.pressure 2>/dev/null

# watch throttle events live
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @[args->rwbs] = count(); }'

A rising full avg60 under steady load is the clearest evidence you have outgrown the plan’s IOPS allocation. That is a commercial problem, not a tuning problem.

Filesystem and journal mode

On ext4, data=ordered is the safe default, but for a database volume that already performs its own crash recovery (PostgreSQL, MySQL with innodb_flush_method=O_DIRECT), data=writeback removes a large serialization point. Reducing journal commit frequency also collapses write amplification:

# /etc/fstab options that matter
/dev/vdb /var/lib/mysql ext4 noatime,nodiratime,data=writeback,nobarrier 0 2

# reduce commit interval (default 5s -> 60s on battery-backed or replicated storage)
mount -o remount,commit=60 /var/lib/mysql

# XFS alternative: separate log device, larger allocation groups
mkfs.xfs -f -l size=128m,version=2 -d agcount=8 /dev/vdb

Measure the delta rather than trusting the theory: run the mixed 70/30 fio job before and after each change and record p99. On a typical 4-vCPU VPS, switching a database volume from data=ordered with a 5-second commit to writeback with a 60-second commit reduces 16k mixed p99 from ~2.4 ms to ~0.9 ms. If you are evaluating whether managed platform defaults already do this for you, compare managed and unmanaged offerings at virtualserversvps.com before spending hours on hand tuning.

Step 3: Confirm With Continuous Data

Single benchmarks are snapshots. Track latency continuously so you can distinguish a permanent ceiling from a noisy-neighbour episode. iostat extended output gives await percentiles indirectly through the queue-length relationship; bpftrace gives them directly from the block layer, including requests that never complete because they were throttled:

iostat -x 5 /dev/vda | awk '{print strftime("%T"), $1, $10, $11, $21, $22}'

# histogram of block I/O completion latency, all devices
sudo bpftrace -e 'tracepoint:block:block_rq_complete {
  @ns = hist(args->nr_sector > 0 ? nsecs - @start[args->dev, args->sector] : 0);
  @start[args->dev, args->sector] = nsecs; }'

Persist the fio JSON output to a time-series database and alert on p99 regression rather than on raw IOPS. A disk that slowly loses 30% of its p99 headroom over two weeks is a signal to migrate before an incident, not after.

Checklist

• Always report p50/p99/p99.9 with percentile_list; averages hide throttling.
• Match iodepth to the virtio queue tags, not to a number copied from a bare-metal guide.
• Test none, mq-deadline, and kyber on your actual device and keep the winner in udev.
• Read /proc/pressure/io before blaming the provider; a rising full average means quota, not media.
• Re-measure after every fstab or mount-option change — assume nothing.

Storage tuning is iterative measurement, not a configuration you copy. Establish a reproducible fio profile, change one variable at a time, and keep the JSON output as your regression baseline.