Benchmarking VPS Disk IOPS with fio: Queue Depth vs Latency

A VPS advertised at “10,000 IOPS” will deliver 900 IOPS at queue depth 1 and 9,800 at queue depth 32 on the same disk. That is not marketing deception so much as a definition problem: IOPS is meaningless without a queue depth, block size and read/write mix attached to it. If you benchmark with a single-threaded dd you measure latency, not throughput, and conclude the storage is far worse than it actually is—or the reverse, if you use fio --iodepth=64 for a workload that is really single-threaded database commits.

This is a practical procedure for measuring the three numbers that matter on a virtual disk—IOPS at a stated queue depth, p99 latency, and the point at which latency collapses—using fio with accurate job files. It also covers what a hypervisor does to those numbers and how to tell a real limit from a noisy neighbour.

Why dd and Single-Thread Tests Mislead

dd issues one synchronous write and waits for the completion. With 4 KiB blocks, that measures round-trip latency converted into a rate: at 0.5 ms per I/O you get roughly 2,000 IOPS, but the number reflects queue depth 1 only. Most production storage accumulates outstanding requests—a database with several writer threads, nginx writing logs from many workers, or a container runtime unpacking layers.

At any queue depth, a virtual disk’s service time is the sum of the guest-side submission, the virtio path, the host filesystem or LVM layer, and the physical device. That extra hop does not change throughput much under load, but it can double p99 latency compared with bare metal. Measure both the rate and the tail, or the rate will hide a latency cliff that users feel as stalls.

Building a fio Job That Answers a Question

Each job should isolate one variable. Start with a random 4 KiB read test at depth 1, then depth 32, on a dedicated scratch file. Avoid benchmarking on the root filesystem of a live instance: background writes from the OS distort the result.

[global]
filename=/mnt/scratch/fio-test
direct=1
ioengine=libaio
bs=4k
size=2G
time_based=1
runtime=60
ramp_time=10
gtod_inactive=1

[randread-qd1]
rw=randread
iodepth=1

[randread-qd32]
rw=randread
iodepth=32
numjobs=4

Create the test file once with fio --name=fill --filename=/mnt/scratch/fio-test --size=2G --bs=1M --rw=write --direct=1 so the random-read jobs operate on a fully allocated file instead of sparse zeros. direct=1 bypasses the page cache, which is essential—cached reads measure RAM, not disk.

Run with --output-format=json and keep the raw output. The summary lines report both IOPS and clat percentiles, but the JSON contains the full distribution and lets you compare runs later without re-benchmarking.

Reading IOPS, BW and Latency Percentiles

The numbers to extract are iops, bw, clat_ns.percentile.99.000000 and clat_ns.percentile.99.900000. Average latency is the least useful statistic; a mean of 0.4 ms with a p99.9 of 40 ms describes a disk that stalls under pressure even though it looks fast on paper.

Workload	Depth	IOPS	Mean lat	p99 lat
4k randread	1	1,900	0.52 ms	1.1 ms
4k randread	32	18,400	6.9 ms	14.2 ms
4k randwrite	32	7,100	17.9 ms	62.0 ms
1M seqwrite	8	640 MB/s	12.5 ms	28.4 ms

The pattern in that table is typical: IOPS scales roughly linearly to the device’s limit, latency grows with depth, and random writes are far slower than reads on the same virtual disk. A 4 KiB random write at depth 32 costing 18 ms is entirely normal for a network-backed volume; it is not a fault.

Finding the Latency Knee

Every device has a point where adding queue depth stops adding IOPS and starts adding latency. Find it by sweeping depth in one job file with iodepth_batch_submit left at default, or by running a short series:

for d in 1 2 4 8 16 32 64 128; do
  fio --name=qd$d --filename=/mnt/scratch/fio-test --direct=1 \
      --ioengine=libaio --rw=randread --bs=4k --size=2G \
      --iodepth=$d --runtime=20 --time_based --ramp_time=5 \
      --gtod_reduce=0 --output-format=json \
      --output=/root/fio-qd$d.json
done
python3 - <<'EOF'
import json, glob
for f in sorted(glob.glob('/root/fio-qd*.json'), key=lambda x: int(x.split('-qd')[1].split('.')[0])):
    j = json.load(open(f))['jobs'][0]['read']
    print(f.split('-qd')[1].split('.')[0], j['iops'], j['clat_ns']['percentile']['99.000000']/1e6)
EOF

Plot IOPS against depth. The knee is where each additional unit of depth yields less than roughly 10% more IOPS while p99 latency rises. For interactive services, run below the knee; for batch jobs such as a nightly index rebuild or a backup, running above it is fine because throughput matters more than tail latency.

Telling a Real Limit from a Noisy Neighbour

A consistent ceiling with stable p99 across repeated runs is a provisioned limit. Erratic results—IOPS varying by 2x between identical runs, or p99 spiking to 100 ms without a matching load increase—suggest contention from another tenant or a saturated host. Repeat the depth-32 test five times and compare medians, not single runs.

Also check iostat -x 1 during the run for %util, await and aqu-sz. If %util sits at 100% while IOPS is flat, the virtual device is saturated; if await climbs but %util stays low, the bottleneck is upstream of the guest. For a more direct signal on steal, compare wall-clock runtime to the sum of CPU time consumed by the job.

Testing the Writes That Actually Hurt

Random reads are the easy case. Filesystems that must flush metadata—ext4 with data=ordered, XFS with journaling, or any workload issuing small synchronous writes—are dominated by write latency and fsync cost. Test with a sync write job and a dedicated fsync job, because they separate device throughput from filesystem journal behaviour.

fio --name=syncwrite --filename=/mnt/scratch/fio-sync \
    --size=512M --bs=4k --rw=randwrite --ioengine=psync \
    --fdatasync=1 --runtime=60 --time_based --direct=1

# measure fsync latency directly
fio --name=fsync --filename=/mnt/scratch/fio-sync \
    --size=512M --bs=4k --rw=write --ioengine=sync \
    --fsync=1 --runtime=60 --time_based --direct=1 \
    --latency_target=5ms --latency_percentile=99

--latency_target changes the test from "what does the disk do" to "what IOPS can it sustain while keeping p99 under 5 ms"—a far more relevant number for a transactional database. The reported result is the sustainable rate at that target, which is usually well below the peak IOPS figure.

Benchmark Hygiene and Repeatability

Always benchmark on a freshly provisioned instance before it carries real data, and record kernel version, filesystem, scheduler and the storage backend. Virtual disk performance changes with the host's load, so a benchmark taken at 03:00 and one at 14:00 on the same instance may differ by 30%. Keep the run window and the raw JSON.

Finally, delete the scratch file afterwards—a 2 GiB test file left on a 40 GiB disk is an operational hazard—and remember that a benchmark describes queue depth at one point in time, not your application. Correlate it with real latency from iostat or application-level metrics before drawing conclusions about whether a change is needed. If the workload is I/O-heavy enough that the queue-depth ceiling is a daily constraint, the platform choice matters more than tuning; our VPS vs dedicated server comparison walks through where local NVMe changes the picture. For the general case, understanding what is guaranteed on shared storage is worth reading before sizing an instance—see the cloud VPS benefits breakdown.