KVM VPS Performance: Steal Time, virtio vs Emulated Devices, and What You Can Tune

A KVM guest that reports 100% CPU usage and delivers 60% of the throughput of an identical bare-metal host is telling you two different things at once. The first is that the hypervisor took time away from your vCPU, which appears as steal time. The second is that the guest is driving emulated hardware where a paravirtual driver would have been used, which appears as elevated system time per I/O operation. Both look like “the server is slow” and both have concrete counters attached.

Steal time is the single most useful number on a VPS because it cannot be hidden: /proc/stat reports it per CPU, and a sustained value above 5% on a production guest means the pCPU is oversubscribed or your neighbour is running a compilation loop. This article separates the two failure modes, gives thresholds, and lists what you can actually change from inside a guest. If you have not yet picked a provider, check which ones publish CPU allocation models on our provider comparison list before chasing kernel knobs.

Reading Steal Time Correctly

The steal field in /proc/stat is the eighth column, measured in USER_HZ ticks of time during which the vCPU was runnable but the hypervisor scheduled something else. It is accounted per-vCPU, so an 8-vCPU guest can show 0% on six CPUs and 40% on two, which is common when the hypervisor pins your vCPUs unevenly or when one process is bound to a single thread.

Aggregate numbers hide this. Always read per-CPU, over a window long enough to smooth the 100 Hz tick, and correlate with the application’s own latency. A 2% average that is 0% for 50 seconds and 8% for 10 seconds is not the same system as a steady 2%.

# per-vCPU steal, 5 samples at 2s
mpstat -P ALL 2 5 | awk 'NR==1 || /^[0-9]/ {print $2, $3, $4, $6, $9}'

# raw ticks: fields 1..10 of each cpu line, 8th is steal
awk '/^cpu/ {print $1, "steal_ticks=" $9}' /proc/stat

# steal since boot as a percentage of total
awk '/^cpu / {t=$2+$3+$4+$5+$6+$7+$8+$9; print 100*$9/t}' /proc/stat

Field numbering varies between kernel versions and tooling. Verify by diffing two reads a second apart rather than trusting a blog post, including this one: the steal column is the one that increases while idle does not.

virtio versus Emulated: The Second Failure Mode

An emulated disk or NIC means every I/O operation traps into the hypervisor, which decodes the instruction, emulates the device register write, and returns. A single 4 KiB random read on an emulated IDE controller can cost 10-20 microseconds of hypervisor time; the same read through virtio-blk or virtio-scsi costs roughly 1-3 microseconds on the same host. Multiply by 20,000 IOPS and the difference is a full core.

The tell is not steal time but system time per operation, plus the device names. /dev/sda through /dev/sdz on a guest can be either virtio or SATA; /dev/vda is virtio, and anything under /sys/block/*/device/model reading QEMU HARDDISK is emulated.

lsblk -o NAME,TRAN,SIZE,MODEL
cat /sys/block/vda/device/model 2>/dev/null
ethtool -i eth0 | grep -E 'driver|version'
lspci | grep -iE 'virtio|ethernet|sata'
dmesg | grep -iE 'virtio_blk|virtio_net|ata[0-9]'

Device	Typical cost per 4K read	Guest system time at 20k IOPS
virtio-blk (no iothread)	1-3 µs	2-6% of a core
virtio-scsi with iothread	2-4 µs	4-8%
emulated AHCI/SATA	10-20 µs	20-40%
emulated IDE	15-30 µs	30-60%

When Emulation Appears Without a Compromise

Some guests show a virtio disk and still pay emulation-like costs in the network stack. The reason is usually that the virtio-net device is running without multi-queue, so every packet is handled on a single vCPU regardless of how many you have. Check the queue count and whether the guest driver negotiated queue pairs.

The other common case is virtio-balloon pressure. When the host needs memory back it inflates the balloon, and the guest’s page allocator starts reclaiming hot cache to satisfy what looks like normal pressure. That produces latency spikes that have nothing to do with CPU at all — and no steal time, which is what makes it confusing.

ethtool -l eth0                 # current vs maximum combined channels
ls /sys/class/net/eth0/queues/
cat /sys/devices/system/cpu/online
egrep -i 'balloon|ballooned' /var/log/kern.log

Thresholds Worth Acting On

These are starting points for a guest running a latency-sensitive service, not universal truths. A batch host can tolerate far more. The point is to have a number that triggers investigation instead of a vague sense that things feel slow.

Metric	Healthy	Investigate	Escalate to host
steal time, p95 per vCPU	< 3%	3-8%	> 10% sustained
steal on any single vCPU	< 5%	5-15%	> 20%
system time at 10k IOPS	< 8%	8-20%	> 25%
p99 read latency, 4K random	< 1 ms	1-5 ms	> 10 ms
run queue length	< vCPUs	= vCPUs	> 2x vCPUs

What You Can Tune Inside the Guest

You cannot fix steal time from inside the guest; that is a scheduling decision made by the host. You can reduce how much of it you suffer, by using fewer busy-polling threads and by making sure you do not have more runnable threads than vCPUs. A process pool of 16 workers on a 4-vCPU guest spends most of its time waiting for a CPU it does not have, and steal simply adds to that queue.

Block layer settings are yours to change. Raising the virtio queue depth and switching the I/O scheduler to none for an SSD-backed virtio device reduces per-request overhead measurably, because mq-deadline adds merge and sort work that a fast device does not need. On emulated devices that change is irrelevant — the trap dominates.

echo none > /sys/block/vda/queue/scheduler
cat /sys/block/vda/queue/nr_requests      # 128 default, 256 for NVMe-backed
cat /sys/block/vda/queue/max_sectors_kb
cat /proc/sys/kernel/sched_rr_timeslice_ms

Method Before Tuning Thresholds

Fix the measurement before you change anything. A single mpstat sample means nothing; collect at 10-second intervals for an hour and compare the p95, not the mean. And check cat /proc/pressure/cpu on 5.8+ kernels, which shows how much time tasks spent stalled on CPU rather than simply runnable — the two diverge when steal is present.

Measuring the Difference Properly

To prove an emulated device is costing you, compare the same workload on two guests that differ only in the controller. Run fio with a fixed queue depth and iodepth, capture guest system time from pidstat, and compare microseconds per request rather than total throughput — throughput is usually capped by the storage backend long before the controller matters.

For steal, the comparison is temporal, not spatial: record per-vCPU steal every 10 seconds for an hour, then overlay your application’s p99 latency. If the two curves move together, the latency has a scheduling cause and no amount of guest tuning will fix it. That is the point to ask the provider for a different host, and to check whether the plan’s CPU is dedicated or shared — the VPS feature breakdown lists which tiers state dedicated cores, which is the only durable fix.