CPU Steal on KVM, Xen and VMware: Reading the Counters That Prove It

If ci_steal in a sosreport summary reads above 20 % on every vCPU while your guest load average sits at 0.4, you have a scheduling problem and no application tuning will fix it. This article walks through reading hypervisor CPU accounting from inside a KVM guest — the same counters a host operator looks at, reached through the interfaces exposed to you — and separating three conditions that all feel like “slow CPU”: steal, quota throttling, and memory-side contention. It assumes a standard KVM/QEMU guest with virtio devices and a stock Ubuntu or Debian kernel, which is what most commercial VPS plans ship.

Context matters, because a plan billed as dedicated-core behaves very differently from one sold as shared vCPU, and the measurement below is the fastest way to tell which one you actually bought. If you are still weighing plan shapes, the VPS vs dedicated server comparison covers the economics; what follows is the diagnostic layer underneath that decision.

What steal time actually counts

The steal field in /proc/stat is incremented by the guest when it yields the vCPU back to the hypervisor while it still had runnable work. In QEMU/KVM that maps to time a vCPU thread was not scheduled on a physical host CPU. Three distinct host-side situations produce it:

  • Oversubscription — more runnable vCPU threads than physical cores, so the host scheduler rotates you out. Steal rises with total host load and is roughly even across your vCPUs.
  • Noisy neighbour — another guest bursting into your host CPU pool. Steal becomes spiky and asymmetric.
  • cgroup quota throttling — the host caps your instance with cpu.max, so once you exhaust the period budget the vCPU is parked. That is throttle, not steal, but the guest-visible symptom is nearly identical.

The distinction matters because remediation differs: oversubscription means move plan or accept it, noisy neighbour means file a ticket with evidence, throttling means your workload exceeds the plan’s burst allowance and needs pacing.

Baseline the guest in five commands

Start with per-vCPU resolution rather than the aggregate top shows. Averages hide a single vCPU pinned at 80 % steal while the other three are idle — common on asymmetric oversubscription.

vmstat 5 12
mpstat -P ALL 5 12
awk '/^cpu /{print "steal_ticks="$9}' /proc/stat
grep -E 'se.nr_migrations' /proc/self/schedstat 2>/dev/null
cat /proc/pressure/cpu

/proc/pressure/cpu (PSI) is the most useful single file on modern kernels. It reports the percentage of wall time where some task was runnable but no CPU was available. some avg10 and full avg10 both at 0.00 means you were never starved: your application is slow at 20 % CPU because it waits on something else, usually I/O. A full value above 5 with idle vCPUs is the signature of host-side scheduling starvation. Record ten minutes at idle, then again under your real workload; the delta in full avg60 is the number to argue with your provider about. A single instantaneous sample is worthless.

Telling steal from throttling

Quota throttling shows up in the time-to-run distribution, so measure latency rather than throughput:

for i in $(seq 400); do
  s=$(date +%s%N); sleep 0.001; e=$(date +%s%N)
  echo $(( (e-s)/1000 ))
done | sort -n | awk '{a[NR]=$1} END {
  printf "p50=%dus p99=%dus max=%dus\n", a[int(NR*0.5)], a[int(NR*0.99)], a[NR]}'

On a healthy shared vCPU expect p99 in the 1.5–6 ms range. If p99 lands consistently between 95 and 105 ms, a quota is parking vCPUs until the next period boundary and tasks cluster around it. If the tail is long and unstructured (p99 30 ms, max 400 ms) with asymmetric per-vCPU steal, it is host contention. A second discriminator is whether the effect scales with your own CPU use: run stress-ng --cpu 1 --cpu-load 50 --timeout 120 and watch steal. If steal grows with your usage and plateaus once you hit a fixed level, a quota is binding. If steal appears at 10 % utilisation of one vCPU, you are sharing a busy host.

Reading KVM and container counters without host access

A KVM guest exposes pvclock data, which tells you whether the clock source is stable and whether host frequency scaling is penalising short bursts. The useful identifiers are the clocksource in use and the hypervisor flag reported through CPUID:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
dmesg | grep -i -E 'kvm-clock|tsc.*stable'
grep -m1 flags /proc/cpuinfo | tr ' ' '\n' | grep hypervisor
systemd-detect-virt --vm

systemd-detect-virt returning kvm confirms full virtualisation. On container-based plans it returns lxc or openvz, and there steal is meaningless because the guest shares the host kernel and slice accounting happens in cgroups. Switch to the cgroup v2 files:

cat /sys/fs/cgroup/cpu.stat
# nr_periods / nr_throttled / throttled_usec — throttled_usec over usage_usec
# above 0.05 means you hit a quota in most periods
cat /sys/fs/cgroup/cpu.max   # "200000 100000" = 2 CPUs per 100 ms period

Xen and VMware signatures

Xen guests on a paravirtualised kernel report steal in the same /proc/stat position, but granularity depends on tsc_mode and whether the host runs the credit or credit2 scheduler. Credit2 produces smoother steal; credit produces bursty steal tied to other domains’ credits. If xl info is installed you can confirm the scheduler, but on commercial Xen VPS it usually is not — rely on PSI plus per-vCPU steal instead.

VMware is the awkward case. ESXi guests report CPU ready time as a host-side metric; inside the guest the only symptom is higher scheduler latency, and steal reads flat because the guest clock abstraction does not surface it. If systemd-detect-virt returns vmware while PSI full rises under load, ask the provider for the VM’s cpu.ready.summation. Sustained ready time above 5 % per vCPU is the VMware equivalent of unacceptable steal.

Correlating steal with application pain

Steal only matters if it changes your latency distribution. The test that settles it is a fixed-rate workload whose service time you already know: a static-file load test with hey, run once in a quiet window and once during a suspected contention window, with PSI logged alongside.

hey -z 120s -c 16 http://127.0.0.1/1k.html
while :; do printf '%s ' "$(date +%T)"; grep '^full' /proc/pressure/cpu; sleep 10; done

A rule of thumb from many KVM deployments: each extra percentage point of steal costs roughly 1–2 % on p99 for CPU-bound request handling, because short requests get unlucky in the run queue; I/O-bound requests are far less sensitive. If p99 degrades 3× while steal rises from 2 % to 6 %, the binding constraint is the scheduler tail rather than total CPU, and dedicated cores are the only real fix. Presenting this to a provider works best as four numbers with timestamps: per-vCPU steal, PSI full over 30 minutes, the p99 shift under fixed load, and when each was sampled. Vague “my server is slow” tickets get closed.

Thresholds and what to do at each

Steal (per vCPU, 30-min avg)PSI full avg60InterpretationAction
< 2 %< 1Normal for shared vCPUNone; keep a baseline
2–8 %1–5Oversubscribed but managedProfile before scaling; often acceptable
8–20 %5–20Binding contentionTicket with data; quantify p99 cost
> 20 %> 20Host is oversoldMove to a dedicated-core plan
0 %, PSI flat, latency high0Not CPU — check I/O and memoryGo to disk and page-fault analysis

The last row is the one people miss. A guest with zero steal, zero PSI and high latency is waiting on a block device or on page reclaim, and every CPU-side metric you collect will confirm the CPU is fine. If that is where you land, the managed vs unmanaged VPS trade-off is worth reading before you spend a weekend on kernel tuning — a managed plan absorbs exactly this kind of investigation and separates host-side from application-side causes for you.

Persist the baseline as a time series

Steal is only useful as a series, not a sample. A five-line collector writing one line a minute is enough, and it costs nothing measurable even on a 1 vCPU instance:

#!/bin/sh
# /usr/local/bin/steal-sample
read _ u n s i io si st _ < /proc/stat
psi=$(grep '^full' /proc/pressure/cpu | sed 's/.*avg60=\([0-9.]*\).*/\1/')
echo "$(date +%s) $st $psi" >> /var/log/steal.log

Drive it from a systemd timer at OnUnitActiveSec=60s and rotate weekly. When throughput drops three weeks later, the correlation between the steal series and your application latency tells you in one glance whether to investigate your own stack or the host. That is the point of the exercise: not to complain about steal, but to know precisely when it explains a slowdown — and when you are about to waste a day tuning a sysctl that was never involved. If the split between isolation modes is still unclear, how VPS isolation differs from shared hosting explains why these counters exist at all: on shared hosting there is no guest kernel to measure.

Leave a Reply