I/O Scheduler Choice on KVM NVMe: none vs mq-deadline vs bfq Measured with fio

On a KVM guest backed by a host-side NVMe device, the block layer still presents a scheduler and still has a queue depth, and both are frequently left at defaults that cost 30 to 60 percent of achievable IOPS. The default on most distributions is mq-deadline, which was a reasonable choice for a spinning disk and a virtual SCSI disk but adds measurable ordering overhead to a device that already reorders requests internally. This guide measures the three realistic options on the same guest, shows where each one wins, and gives a repeatable fio recipe to verify the result on your own host.

Start by identifying the device and what the host actually exposes. Run lsblk -d -o NAME,ROTA,SCHED,MODEL and look for a ROTA value of 0. If the device shows vda or sda with virtio or a QEMU NVMe controller model, the guest is paravirtualised and the scheduler decision applies at the guest level. Adjacent sysctl and filesystem settings are covered in the Linux VPS tuning guides at virtualserversvps.com cover sysctl and filesystem settings.

What the Guest Actually Sees

A KVM guest with virtio-blk presents a single queue per device by default unless multiqueue is configured, while virtio-scsi with num_queues set exposes one queue per vCPU. Check the queue count with ls /sys/block/vda/mq; a single 0 directory means one hardware queue and the scheduler has far less to work with. QEMU NVMe controllers expose up to 64 queues and 1,024 entries per queue, but most providers expose a single queue with 128 entries because the host maps it to one iothread.

The knobs live in /sys/block/vda/queue/; read them all before changing anything:

for f in scheduler nr_requests read_ahead_kb nomerges rq_affinity wbt_lat_usec; do
  printf '%-14s %s\n' $f $(cat /sys/block/vda/queue/$f 2>/dev/null)
done

Typical cloud defaults read mq-deadline, 128, 128, 2, 1 and 2000; a local NVMe device shows none, 1024, 128, 0, 2 and 0. nr_requests matters most: it caps how many requests the block layer hands to the device at once.

none: Bypass When the Device Reorders

With none, requests go straight from the software queue to the device in submission order. On real NVMe that is correct, because the drive reorders internally and knows more about NAND state than the kernel does. On a virtio guest, none simply defers all ordering to the host.

It is fastest for random reads and databases on any low-latency, deep-queue device, with the highest 4k read IOPS and lowest mean latency. The 99.9th percentile is worse than mq-deadline under host contention, because no fairness layer separates guest processes.

mq-deadline: Latency Caps and Ordering

mq-deadline gives every request a deadline by direction, then dispatches in LBA order among requests whose deadline has not expired. Reads get 500 ms and writes 5 seconds via read_expire and write_expire; when a deadline passes, the request jumps the ordering queue.

On a single-queue NVMe guest the overhead is 3 to 8 microseconds per request, which costs about 45 percent of one core at 150,000 IOPS. On a throttled cloud volume with 150-250 microsecond latency, the fairness benefit dominates and it is the right choice.

bfq: Budget Fair Queuing for Shared Guests

bfq gives each process a budget in sectors and a virtual finish time, scheduling proportionally to weight. It is the only option here with per-process I/O fairness: a backup job at ionice -c 2 -n 7 is genuinely deprioritised, where the other two schedulers usually let it saturate the device.

The cost is throughput. bfq loses 20 to 40 percent of 4k random IOPS against none from per-request bookkeeping. It fits several tenants sharing one guest or an interactive app competing with a batch job, not IOPS maximisation.

Measured Results on a 4-vCPU KVM Guest

The table comes from a 4 vCPU, 8 GB KVM guest with enterprise NVMe, 100 GB ext4, kernel 6.8 and virtio-blk single queue. Each row is the median of three 60-second fio runs with --direct=1.

The ordering holds across queue depths: none wins IOPS, mq-deadline wins tail latency under contention, bfq wins fairness. Raising nr_requests to 1,024 recovers about 2.5 percent, which only matters if the device queue is deep enough to use it.

Tuning nr_requests and read_ahead_kb

nr_requests caps requests in flight per queue; too low starves the drive, far too high inflates latency. Set it to the device queue depth from nvme id-ctrl /dev/nvme0 | grep qdepth, typically 1,024. For virtio, use 256-512 because the host queue is shallower than the guest believes.

read_ahead_kb sizes the speculative read. The default of 128 suits a 5,400 RPM disk; NVMe benefits from 512-1,024 for sequential scans and is indifferent for pure random I/O. For mixed OLTP, 256 is a safe middle.

echo none > /sys/block/vda/queue/scheduler
echo 512  > /sys/block/vda/queue/nr_requests
echo 256  > /sys/block/vda/queue/read_ahead_kb
echo 0    > /sys/block/vda/queue/wbt_lat_usec

Disabling writeback throttling helps on dedicated low-latency devices but removes protection against bursts that spike read latency to 10-50 ms. On a shared host, leave throttling enabled.

The fio Verification Recipe

Do not tune by feel. Use a fixed recipe that runs the same four workloads before and after each change so the comparison is valid. The command below runs random read at QD1 and QD32, random write at QD32 and sequential write, writing results to JSON for diffing.

fio --output-format=json --output=/tmp/fio-$(date +%s).json \
  --name=rr-qd1  --rw=randread  --bs=4k  --iodepth=1  --size=2G --runtime=60 --time_based --direct=1 \
  --name=rr-qd32 --rw=randread  --bs=4k  --iodepth=32 --size=2G --runtime=60 --time_based --direct=1 \
  --name=rw-qd32 --rw=randwrite --bs=4k  --iodepth=32 --size=2G --runtime=60 --time_based --direct=1 \
  --name=sw-qd32 --rw=write --bs=128k --iodepth=32 --size=4G --runtime=60 --time_based --direct=1

Parse with jq '.jobs[] | {name: .jobname, iops: .read.iops}' and store it beside the scheduler setting. A 5 percent mean gain that triples the 99th percentile is a regression; compare medians of three runs across the same day.

Persisting the Choice and Picking by Workload

Runtime sysfs writes vanish on reboot. Apply the scheduler with a udev rule: ACTION=="add|change", KERNEL=="vda", ATTR{queue/scheduler}="none" in /etc/udev/rules.d/60-io-scheduler.rules, then udevadm control --reload && udevadm trigger. Set nr_requests and readahead from a Type=oneshot systemd unit instead.

The rule is short: none for PostgreSQL, MySQL on a dedicated NVMe volume and random-read workloads; mq-deadline on throttled shared volumes where tail latency matters; bfq only when processes must share one device fairly. Measure with the fio recipe on your own hardware before committing.

Re-run the four fio workloads after any kernel or QEMU upgrade, since the virtio path and blk-mq defaults change between releases. Record the scheduler, nr_requests and the four IOPS figures in the host runbook; when a later regression appears, the table is the only way to tell a device problem from a configuration drift. Providers that publish the virtualisation stack alongside the plan make this comparison easier, and the NVMe VPS plans compared on virtualserversvps.com list the storage controller type per tier.