virtiofs vs 9p on a KVM VPS: Measuring Shared-Volume Latency and Metadata Cost

Some workloads need to read data that lives on the hypervisor, not on a virtual disk: build caches, shared media libraries, a host-mounted dataset you do not want to copy into the guest. On a KVM VPS the traditional option is a 9p virtio-9p share, which is slow but present on almost every kernel. The newer option is virtiofs, a FUSE-based shared filesystem that adds a DAX window for direct access to host memory. Neither behaves like a local NVMe device, and the difference between them is large enough to change what you can run from the share.

This is a measurement guide. It shows how to identify which mechanism is present, what latency each one adds per operation (not per megabyte), and why the obvious benchmark – a 1 GB dd – tells you almost nothing useful about either.

Identify the Share Before You Benchmark It

Check what the guest actually has mounted and how it arrived. Both mechanisms appear as normal mounts, and only the kernel modules and mount options distinguish them:

findmnt -t virtiofs,9p,fuse.virtiofs
mount | grep -E 'virtiofs|9p'
lsmod | grep -E 'virtiofs|9p|fuse'
dmesg | grep -iE 'virtiofs|9pnet'

# virtio-fs device count and tag
ls /sys/bus/virtio/drivers/virtio_fs/ 2>/dev/null

A mount with type 9p and options including cache=none or cache=loose is the legacy path. A mount of type virtiofs with a tag like hostshare is the FUSE-based share. If neither exists, you have no host share and the rest of this article does not apply – note that many VPS products simply do not offer one, and a network volume is the supported alternative.

Why 9p Costs You a Context Switch Per Operation

9p is a message protocol, not a block protocol. Every stat, open, and read is marshalled into a request, delivered through the virtio queue to the host, serviced by QEMU’s 9p server, and answered. With one iothread and cache=none, that is a guest-user to guest-kernel, guest to host, and host-user round trip per operation. Metadata-heavy work collapses: a directory listing of 50,000 files, a git status in a large tree, or a PHP autoloader walking include paths will each take tens of seconds.

Sequential throughput is mediocre but tolerable, and the sync mode matters more than the transport. With cache=loose the guest caches aggressively and throughput improves, at the cost of coherence: two guests or the host seeing different versions of the same file. That is a correctness trade, not a tuning knob.

What virtiofs Changes, and What It Does Not

virtiofs replaces the 9p protocol with a FUSE server running on the host, still over a virtio queue. That removes most of the protocol marshalling cost and adds a shared memory window (DAX) through which file contents can be mapped directly instead of copied. The win is largest on metadata operations and on mmap’d reads; the win is small on plain sequential writes.

DAX is conditional. It needs the shared memory region exposed by the host, a host-side directory on a filesystem supporting the mapping, and a guest kernel new enough to negotiate it. Check the mount options for dax=always or dax=inode after mounting; if it is absent, you are on the copy path and should expect roughly FUSE-copy performance, not memory speed.

findmnt -no OPTIONS /mnt/share
# look for dax=always / dax=inode
cat /sys/kernel/debug/virtiofs/* 2>/dev/null | head

The Benchmark That Actually Discriminates

Sequential dd measures the transport and hides the protocol. Use fio with small-block random reads at queue depth 1 for latency, then a metadata storm for protocol overhead. Run both against the share and against a local directory on the same guest to get the ratio, since absolute numbers vary by host.

# per-operation latency, 4k random read, QD1
fio --name=share-lat --directory=/mnt/share --rw=randread --bs=4k \
    --iodepth=1 --size=256M --direct=1 --runtime=30 --time_based \
    --output-format=json --output=/tmp/share-lat.json

# metadata storm: create and stat 20,000 empty files
mkdir -p /mnt/share/mdtest
time ( seq 1 20000 | xargs -P 4 -I{} touch /mnt/share/mdtest/{}.tmp )
time ( seq 1 20000 | xargs -P 4 -I{} stat -c %s /mnt/share/mdtest/{}.tmp >/dev/null )

# control: same on local disk
fio --name=local-lat --directory=/root --rw=randread --bs=4k \
    --iodepth=1 --size=256M --direct=1 --runtime=30 --time_based \
    --output-format=json --output=/tmp/local-lat.json

Compare .jobs[0].read.clat_ns.percentile["99.000000"] between the two files, not the mean. The p99 is what makes a share feel broken in production, and it is where 9p and virtiofs diverge most.

for f in /tmp/share-lat.json /tmp/local-lat.json; do
  jq -r --arg f "$f" '.jobs[0] as $j | "\($f) iops=\($j.read.iops) p99=\($j.read.clat_ns.percentile["99.000000"]/1000)us"' $f
done

Interpreting the Ratios

MetricLocal NVMe9p (cache=none)virtiofs (no DAX)virtiofs (DAX)
4k randread QD1 IOPS12,000-20,0001,500-4,0008,000-15,000nearly local
p99 read latency0.1-0.3 ms1-8 ms0.3-1 ms0.2-0.5 ms
20k file create+stat2-4 s60-180 s6-15 s6-15 s
1 GB sequential readline rate300-800 MB/s600-1,500 MB/s2,000 MB/s+

The pattern to take away: 9p is roughly an order of magnitude worse on metadata and 4-10x worse on latency, while sequential throughput looks almost respectable. That gap is why “the share is fast, it copies at 700 MB/s” and “the share is unusable, my build takes ten minutes” are both true statements about the same mount.

What Each Share Is Good For

Both are host-mediated, so neither survives the host rebooting without you, and neither is a substitute for a backed-up volume. Decide by operation mix:

  • Read-only large files, mmap’d – virtiofs with DAX. Caches, dataset images, media. This is the case virtiofs was designed for.
  • Read-only large files, no DAX – virtiofs still wins, but expect copies through the page cache on every read.
  • Build outputs, databases, mail spools – neither. Use a real volume. FUSE will dominate your latency profile and every fsync becomes a host round trip.
  • Legacy kernel, no virtiofs module – 9p with cache=loose and msize=512k is the only option; budget for the metadata cost and keep the tree small.

For anything writable and performance-sensitive, a block volume attached to the guest is cheaper than tuning a share that was never built for fsync.

A Measurement Checklist

Confirm the mount type and DAX status, run QD1 random reads for p99 latency, run a 20,000-file create and stat sweep, then repeat every number against local disk on the same guest. Publish the ratio, not the raw figure, because the host underneath changes every migration. If DAX is available and the workload is read-mostly, virtiofs is close enough to local that most applications will not notice; if it is 9p, design around the metadata cost rather than trying to tune it away.

Whether a shared volume is even the right answer depends on the plan and the platform behind it. The VPS plans overview sets out which tiers include attachable block volumes, and the cloud VPS benefits page explains the storage architecture that decides whether virtiofs, 9p or a local block device is the sane choice.

Leave a Reply