Reading Hypervisor CPU Steal: KVM, Xen, and VMware Counters Without a Host Login

When a VPS feels slow but top shows idle CPU and free memory, the problem is usually outside your guest — in the hypervisor scheduler. This article shows how to extract hypervisor-level CPU accounting from inside a guest, distinguish steal from throttle from contention, and interpret the platform-specific counters that KVM, Xen, and VMware each expose differently. No host access required.

What Steal Time Actually Counts

%steal is the time your vCPU was runnable but the physical CPU was executing another task — either another guest’s vCPU or host kernel work. It is not the same as being throttled (where the hypervisor voluntarily stops your vCPU after you exhaust a quota), and it is not the same as waiting on I/O. Conflating the three leads to the wrong remediation. Read the raw counters first:

# per-CPU breakdown including steal and guest
mpstat -P ALL 2 5

# steal only, one number per second per CPU
grep '^cpu[0-9]' /proc/stat | awk '{print $1, $9}'

# verbose: guest, guest_nice, steal in USER_HZ
cat /proc/stat | head -1
# cpu  user nice system idle iowait irq softirq steal guest guest_nice

# derive a live percentage (steal is field 9)
awk '/^cpu /{t=$2+$3+$4+$5+$6+$7+$8+$9+$10+$11;
  printf "steal=%.2f%%\n", $9/t*100}' /proc/stat

Turning Counters Into a Diagnosis

Sample steal over a window and correlate it with your own run-queue length. The combination tells you which of three problems you have:

Steal	Guest run-queue	iowait	Diagnosis
<2%	Low	Low	Healthy; look at application code
>10%, steady	Low	Low	Host CPU oversubscription; plan-level issue
>10%, bursty	High	Low	Noisy neighbour on the same physical core
High	Low	High	Storage quota, not CPU — re-check I/O pressure
Spiky, periodic	Low	Low	Host housekeeping, backups, live migration

Get a continuous, low-overhead series rather than eyeballing top:

# 1-second samples to CSV for 10 minutes, then summarise percentiles
sar -u ALL 1 600 > /tmp/cpu.txt 2>&1
grep -v '^Linux\|^Average\|^$' /tmp/cpu.txt | awk '{print $NF}' \
  | sort -n | awk '{a[NR]=$1} END{
      print "p50 steal:", a[int(NR*0.5)], "p99:", a[int(NR*0.99)], "max:", a[NR]}'

# per-vCPU steal distribution, which matters more than the average
for c in /sys/devices/system/cpu/cpu[0-9]*; do
  echo -n "$(basename $c): "; awk '{print $9}' $c/stat 2>/dev/null
done

Per-vCPU distribution is the key insight. A 4-vCPU guest where vCPU0 shows 25% steal and the others show 0.5% has a single pinned physical thread contended by a neighbour; a guest where all four sit at 8% has a plan that is simply oversold. The first is often fixable by asking the provider to rebalance; the second requires a different plan — the plan structures at virtualserversvps.com make the dedicated-vCPU distinction explicit.

Platform-Specific Counters

KVM / QEMU: steal plus steal_clock skew

KVM guests expose steal directly through pvclock. The authoritative source is the per-CPU steal field already shown above, but you can also detect vCPU preemption precisely through the paravirt steal-time interface, which the guest kernel uses to stop its own run-time clock while descheduled:

# is steal-time reporting enabled by the host? (bit 1 of flags)
grep -m1 flags /proc/cpuinfo | grep -o 'steal_time'

# halt-polling / preemption effects visible in the halt exit counters
perf stat -e kvm:kvm_steal_time -a sleep 5 2>/dev/null || true

# KVM-specific: check whether the host is using a dedicated thread per vCPU
ls /sys/devices/system/cpu/cpu0/thermal_throttle/ 2>/dev/null
cat /sys/kernel/debug/kvm/*/halt_exits 2>/dev/null | head

If /proc/cpuinfo lacks steal_time, the host is not reporting preemption, which itself is informative: many budget providers disable it because it makes their oversubscription visible.

Xen: the runstate domains

Xen splits the concept into explicit run states, which is more precise than KVM’s single steal counter. RUNNING means executing, RUNNABLE means ready but not scheduled — that is your steal — and BLOCKED means waiting on I/O or idle. Sum these from /proc/xen/privcmd-adjacent interfaces or the xen sysfs, and via xentop if available:

# Xen exposes times through the canonical runstate interface
cat /proc/stat | grep -i steal        # guest-side view, same units
cat /sys/hypervisor/type 2>/dev/null  # should print "xen"

# toolstack view inside the guest, if xenstore is exposed
xentop -b -i 2 -n 1 2>/dev/null | head -20 || echo "no xenstore access (normal for HVM)"

# authoritative runstate report via the hypercall-backed clock
for i in /sys/devices/system/cpu/cpu*/stat; do
  awk '{print FILENAME, "steal_ticks="$9}' "$i"; done

The practical difference: Xen’s RUNNABLE time is more sensitive to short scheduling delays than KVM’s coarse steal counter, so Xen-based hosts often show contention earlier — useful when you are diagnosing a provider before the problem becomes visible in production.

VMware: cpu.ready and cpu.costop

On ESXi-backed virtual machines the same phenomena have different names. cpu.ready (also called ready time) is the equivalent of steal, and cpu.costop counts time lost to vCPU co-scheduling delays — when one vCPU of a multi-vCPU VM must wait for its sibling to be scheduled. In a guest you rarely get direct access, but the guest-side %steal from mpstat is derived from the same physical preemption, and esxtop on the host is what support will ask for. If your provider runs ESXi, budget for the possibility that a large vCPU count makes costop worse: request fewer, faster vCPUs.

What You Can Actually Fix

Pin your own hot threads. If your workload has one busy thread, distributing it across more vCPUs increases steal exposure. Pin it to a single vCPU with taskset or cgroup cpuset.
Reduce context-switch pressure. Excessive threads amplify scheduling delay; check with pidstat -w 2 and cut worker counts to match vCPU count.
Prefer fewer, guaranteed vCPUs. 2 dedicated cores beat 8 shared ones for latency-sensitive workloads every time.
Document the evidence. A p99 steal series plus per-vCPU distribution is a strong support ticket; “the server is slow” is not.

Instrument It Permanently

# node_exporter already ships steal as node_cpu_seconds_total{mode="steal"}
# alert on a sustained p95 rather than instantaneous spikes
# promql:
#   quantile_over_time(0.95,
#     rate(node_cpu_seconds_total{mode="steal"}[5m])[1h:1m]) > 0.05
# for the raw node_exporter scraping path, see the setup notes at
# https://virtualserversvps.com/ if you are starting from scratch

Steal is not a mysterious metric once you separate it from throttle and I/O wait. Collect it per-vCPU, store percentiles rather than instantaneous values, and you will know within a week whether your provider’s contention is chronic or a one-off neighbour.