Measuring Queue Latency vs Device Latency on a Shared-NVMe VPS: A fio + bpftrace Method

When a VPS feels slow, most operators blame CPU first. On a modern hypervisor that is usually wrong: the erratic latency comes from storage, where your writes join a shared NVMe array serving dozens of tenants. The queue depth in front of that device is invisible to iostat because your guest only sees its own virtual block device. This article shows how to split device service time from queue waiting time so you can tell whether your disk is genuinely slow or merely buried behind other tenants.

Why Guest-Side iostat Lies

Inside a guest, iostat -x reports await as a single number. That number is the sum of time spent waiting in the virtio-blk or virtio-scsi queue plus the time the host spent servicing the request. There is no separate column for queue time. If the host’s backing store is oversubscribed, your await grows even though the physical device health is fine.

The fix is to instrument both ends. Inside the guest you measure time from block-layer submit to completion. On the host (if you have access) you measure time from the physical device receiving the request to completing it. The delta is queue latency. Even without host access, you can bound device latency by running a controlled test when the array is quiet and comparing it to production numbers.

Step 1: Baseline with fio Using Latency Percentiles

Never trust a bare average. Use fio with percentile reporting and a fixed iodepth so the measurement is reproducible.

fio --name=seqwrite --filename=/var/tmp/fio.test --size=4G \
  --rw=randwrite --bs=4k --iodepth=32 --ioengine=libaio --direct=1 \
  --runtime=60 --time_based --percentile_list=50:95:99:99.9 \
  --latency_target=0 --output-format=json --output=/tmp/fio.json

jq '.jobs[0].write | {iops, lat_ns: {p50: .clat_ns.percentile["50.000000"],
     p99: .clat_ns.percentile["99.000000"]}}' /tmp/fio.json

The key field is clat_ns (completion latency) versus lat_ns (total latency). Total latency includes queueing; completion latency measures only how long the request took once the driver submitted it. The difference between the two is your queue wait.

Step 2: Trace the Block Layer Directly

On kernels 5.8+ you can attach bpftrace to the block tracepoints and get a live histogram of where time is spent.

bpftrace -e '
  tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
  tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
    @lat = hist(nsecs - @start[args->dev, args->sector]);
    delete(@start[args->dev, args->sector]);
  }
  interval:s:30 { print(@lat); clear(@lat); }'

A bimodal histogram – one tight cluster plus a long tail – is the signature of queue contention. A single wide cluster means the device itself is the limit.

Step 3: Interpreting the Numbers

Metric	Quiet Array	Loaded Array	Interpretation
clat p50	0.28 ms	0.31 ms	Device is fine
clat p99	0.9 ms	7.8 ms	Tail is device or queue
lat – clat p99	0.1 ms	6.9 ms	Queue wait dominates
IOPS at iodepth=32	62k	18k	Throughput starved

When the queue-wait delta exceeds completion latency, no amount of guest-side tuning helps. You are competing for a physical resource you do not control.

Separating Write and Read Paths

Writes and reads behave differently behind a shared array, and measuring them together hides the answer. Reads are frequently served from the host page cache, so they look fast even when the backing store is congested. Writes must survive, so they always traverse the real queue. Run the two directions separately and compare p99 rather than mean.

# read path
fio --name=randread --filename=/var/tmp/fio.test --rw=randread --bs=4k \
  --iodepth=16 --direct=1 --runtime=30 --time_based \
  --percentile_list=50:99:99.9 --output-format=json --output=/tmp/r.json
# write path
fio --name=randwrite --filename=/var/tmp/fio.test --rw=randwrite --bs=4k \
  --iodepth=16 --direct=1 --runtime=30 --time_based \
  --percentile_list=50:99:99.9 --output-format=json --output=/tmp/w.json
jq -n --slurpfile r /tmp/r.json --slurpfile w /tmp/w.json \
  '{read_p99: $r[0].jobs[0].read.clat_ns.percentile["99.000000"],
    write_p99: $w[0].jobs[0].write.clat_ns.percentile["99.000000"]}'

If read p99 sits under one millisecond while write p99 is in the tens, the queue is write-saturated. That is often caused by something inside your own guest rather than a neighbour: a database flushing too aggressively, or a logging daemon calling fsync per line. Check with pidstat -d 1 to attribute the write IOPS to a process before blaming the host.

Establishing a Same-Hour Baseline

Storage latency on a shared array varies by a factor of three over a day. A baseline captured at 03:00 tells you nothing about behaviour at 20:00. Capture at several points and keep the results in a table so you can see the pattern rather than a single misleading sample.

for h in 03 09 15 21; do
  echo "=== $h:00 ==="
  fio --name=t --filename=/var/tmp/fio.test --rw=randwrite --bs=4k \
    --iodepth=16 --direct=1 --runtime=20 --time_based \
    --output-format=json --output=/tmp/fio.$h.json 2>/dev/null
  jq -r '"\"'$h' p50=\(.jobs[0].write.clat_ns.percentile["50.000000"]) p99=\(.jobs[0].write.clat_ns.percentile["99.000000"])"'" /tmp/fio.$h.json
done

What You Can Actually Do About It

• Reduce queue pressure: lower write concurrency by batching (for example, innodb_flush_neighbors=0 plus a larger innodb_log_file_size) so the guest submits fewer, larger requests.
• Increase the guest queue depth: set nr_requests and use a multi-queue scheduler such as none for NVMe-backed devices.
• Move to a provider with dedicated I/O: sustained high queue latency across different times of day is a tenancy problem, not a config problem. Compare plans on our VPS comparison table where storage overcommit is listed per plan.
• Cache aggressively: a page cache and application-level cache remove reads entirely, which is cheaper than fixing latency. See the VPS guides hub for caching layers.

Measurement Discipline

Run the fio baseline at the same hour of day as your production incident, and run it twice: once with the guest idle and once under real load. Only the second number matters for capacity planning. Record clat and total lat side by side; the divergence is your queue tax, and it is the single most useful number for deciding whether to tune or to move. If you find the queue is the problem, the practical path is documented in our VPS comparison table – pick a plan with guaranteed IOPS before spending another week on sysctl.