Measuring CPU Steal Time Accurately on a KVM VPS

Steal time is the one CPU metric on a VPS that describes something you cannot fix from inside the guest. It counts the cycles during which your vCPU was runnable but the hypervisor scheduled something else on the physical core. Almost every guide stops at “look at the %st column in top“, which is where the measurement usually goes wrong: the number depends on the sampling interval, on which CPU you read, and on whether the guest kernel can even see the delay.

The Field Semantics in /proc/stat

Each line in /proc/stat beginning with cpu is a cumulative tick counter. The first line, cpu, is the aggregate across all vCPUs; cpu0, cpu1 and so on are per-vCPU. The fields are user nice system idle iowait irq softirq steal guest guest_nice. There is no field for hypervisor overhead you did not observe — steal is only what the guest kernel was told, via paravirtualised clocks, while it had a runnable task waiting.

The unit is USER_HZ ticks, almost always 1/100 s on x86. Do not treat the values as percentages: they are counters, and the only meaningful number is a difference over a known interval. irq and softirq are accounted separately, and on a modern kernel guest time is already included in user, so subtracting fields naively double-counts. The invariant to watch is that the ten fields sum to the elapsed interval in ticks.

Reading It Correctly: Aggregate vs Per-vCPU

The aggregate cpu line is an average across vCPUs, and averaging destroys the signal you care about. A single pegged vCPU with heavy steal, next to three idle vCPUs, shows up as a few percent on the aggregate and is invisible. Always inspect per-CPU lines when diagnosing latency.

awk '/^cpu[0-9]/{printf "%s steal=%s user=%s sys=%s idle=%s\n", $1,$9,$2,$4,$5}' /proc/stat
mpstat -P ALL 5 3          # per-CPU, 5 s interval, 3 samples
pidstat -u 1 -t            # which threads are consuming the CPU

mpstat takes a delta internally and is far less error-prone than eyeballing top. Its %steal column is computed the same way you should compute yours by hand. If %steal is consistently above a few percent on a vCPU while your own %user and %sys on that core are low, the core is waiting on the host. If %steal is high and %user is also high, you are simply busy and the host is still managing to run you — that is normal framing.

Sampling Interval Pitfalls

Short intervals produce unusable numbers. At a 1-second interval with 100 Hz ticks, each vCPU’s steal counter can only move in units of 0-100 ticks, and a single scheduling delay of 20 ms shows up as 2 percent on one sample and 0 percent on the next. Averaging across 1-second samples systematically understates bursty contention and overstates smooth contention.

Use an interval of at least 10 seconds, and take several samples rather than one. Also be aware of the reverse error: read too far apart and you measure an average including periods when the host was quiet. For latency-sensitive diagnosis, correlate steal with your own request latency on the same timebase rather than summarising both separately.

# 10-second windows, 30 samples, per-vCPU steal in percent
for i in $(seq 30); do
  awk '/^cpu[0-9]/{s[$1]=$9; t[$1]=$2+$3+$4+$5+$6+$7+$8+$9}
       END{for(c in s) printf "%s %s\n", c, s[c]}' /proc/stat > /tmp/a
  sleep 10
  awk '/^cpu[0-9]/{s[$1]=$9; t[$1]=$2+$3+$4+$5+$6+$7+$8+$9}
       END{for(c in s) printf "%s %s\n", c, s[c]}' /proc/stat > /tmp/b
  paste /tmp/a /tmp/b | awk '{d=$4-$2; printf "%s %.1f%%\n", $1, d*100/(10*100)}'
done

Adjust the divisor to your actual getconf CLK_TCK and interval. The script above assumes 100 Hz and a 10-second sleep; both are correct on stock x86_64 images but worth confirming on unusual kernels or architectures.

What Steal Does Not Tell You

Steal accounts for time the guest knew it was runnable and not running. It does not capture cache and memory-bandwidth interference from co-tenants: another VM thrashing the last-level cache makes your instructions slower while your vCPU is scheduled the whole time. That shows up as elevated cycles-per-instruction and lower throughput at unchanged %user, not as steal.

It also does not capture I/O time spent waiting on a shared storage backend, which lands in %iowait and, on a VPS, often reflects a queue on the host rather than your own disk. When steal looks clean but latency is bad, check %iowait, the block-layer queue depths in /sys/block/*/stat, and the per-process delay accounting in /proc/<pid>/stat.

grep -E '^(cpu|ctxt|procs_running)' /proc/stat
cat /proc/pressure/cpu          # PSI: some/full avg10/avg60/avg300
iostat -xz 5 3                  # await and aqu-sz per device

Pressure stall information is the best complement to steal: full in /proc/pressure/cpu means every runnable task was stalled, which is what a latency-critical service actually experiences. Steal explains part of that stall; PSI measures the consequence.

Correlating Steal With Latency

The decisive test is whether latency degrades when steal rises. Log both on the same wall clock. Sample steal every 10 seconds into a file, and have your application log response times with timestamps; then join the two by minute and look at the correlation. If p99 latency doubles in the minutes when steal is elevated and returns when it falls, the mechanism is established.

# minimal logger: epoch, per-CPU steal ticks
while :; do
  awk -v t=$(date +%s) '/^cpu[0-9]/{print t, $1, $9}' /proc/stat
  sleep 10
done >> /tmp/steal.log

# and on the app side, log $request_time or equivalent per request,
# then bucket both by minute and compare medians

An important control: run the same correlation on a quiet period and on a busy period. A correlation that holds only during your own peak hours may simply be your workload, not the host. Comparing your peak against the host’s behaviour at a time when your load is flat is the cleanest way to separate the two.

Deciding Whether You Are Being Throttled

There are two distinct mechanisms and they look similar from top. CPU quota throttling — from a cgroup CPU limit set by the provider — appears as the vCPU being forcibly descheduled once its budget is spent inside each period. You can read it directly in the guest if cgroup v2 statistics are exposed: cpu.stat reports nr_throttled and throttled_usec.

cat /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/cpu.max
# per-process cgroup if your service runs in a slice
cat /sys/fs/cgroup/system.slice/<unit>.slice/cpu.stat

If throttled_usec climbs steadily and cpu.max shows a finite quota, you are being throttled by your own plan’s limit rather than by co-tenant contention, and steal may remain near zero. The two diagnoses lead to different actions: throttle means the plan’s CPU allowance is too small for the workload, while steal means the host is oversubscribed. Sampling both counters for a few minutes distinguishes them reliably.

Reporting the Finding to a Provider

If you conclude the host is oversubscribed, a report with numbers gets a better response than “the server is slow”. Include the per-vCPU steal figures over a defined interval, the corresponding PSI full values, the timestamps in UTC, and the measured latency degradation. That evidence is reproducible and hard to dismiss, and it also tells you whether the issue is transient or structural.

If it is structural and the workload is production, the fix is a plan with dedicated or guaranteed CPU rather than further measurement — the trade-offs between shared and dedicated allocations are covered in our comparison of VPS plans and their CPU guarantees. Before moving, it is worth confirming the guest can see steal at all: some hypervisor configurations report it unreliably, which is one of several signals that the instance is not a good fit, discussed alongside the practical limits of free and low-cost VPS tiers.

A Repeatable Measurement Checklist

• Read per-vCPU steal from /proc/stat, never the aggregate line alone.
• Use 10-second or longer intervals and take at least 30 samples.
• Compute percentages from counter deltas, not from single readings.
• Accompany steal with /proc/pressure/cpu and %iowait to see the consequence.
• Check cgroup cpu.stat for throttling before blaming the host.
• Correlate against application p99 latency on the same timebase.

Accurate steal measurement is mostly about refusing to trust a single number from a one-second sample. Read the counters as deltas per vCPU, pair them with a stall metric, and separate quota throttling from host contention before deciding whether to tune or to change plans.