Every slowdown triage on a VPS eventually forks into the same question: is the box waiting on the disk, or is it running and simply not getting CPU from the hypervisor? Both show up as “the site is slow” and both raise load average, but they have opposite remedies — one wants a query fix or a storage upgrade, the other is a provider conversation. top showing %wa and %st is not enough to decide, because those columns are fractions of a sampling window and hide whether the blocked time is on the application’s behalf or the host’s.
The method below takes about ninety seconds on a live incident and produces a defensible answer: steal share, blocked-task share, per-device service time, and whether I/O the instance issued is actually completing at the rate the provider promised. If you have already established the baseline for your instance’s storage tier — see the VPS disk latency tier verification procedure — the comparison is direct.
Step 1: one vmstat sample tells you which side to pursue
Run vmstat 1 10 during the incident and read the two right-hand columns of blocks that matter: wa (CPU waiting on I/O) and st (time stolen by the hypervisor). The first row is an average since boot and must be discarded. Use the per-second rows.
vmstat 1 10 | tee /tmp/vm.txt
# columns: r b swpd free buff cache si so bi bo in cs us sy id wa st
# decision rule (per-second rows, averaged by eye or awk):
# st > 3 AND wa hypervisor CPU contention
# wa > 15 AND st genuine I/O wait
# both high -> storage backend under pressure (shared LUN)
Step 2: separate steal from real CPU shortage
Steal time is the hypervisor’s own accounting of time your vCPU was runnable but the physical CPU was busy with another guest. It is reported per vCPU in /proc/stat and aggregated by mpstat. A sustained 5–8% on a 2 vCPU instance is roughly a tenth of one core — noticeable, occasionally acceptable. Above 15% on a production instance is a host problem.
# per-vCPU breakdown, 1s intervals
mpstat -P ALL 1 10
# raw steal ticks over a 60s window, converted to percent
awk '/^cpu[0-9]/{print $1}' /proc/stat
a=$(awk '/^cpu /{print $9}' /proc/stat); ad=$(awk '/^cpu /{print $1}' /proc/stat)
sleep 60
b=$(awk '/^cpu /{print $9}' /proc/stat); bd=$(awk '/^cpu /{print $1}' /proc/stat)
echo "steal% = $(echo "scale=2; 100*($b-$a)/($bd-$ad)" | bc)"
Step 3: attribute I/O wait to a device and a process
If wa is high, the next question is whether the device is busy or merely slow. %util near 100 with rising await is saturation; %util at 30 with 40 ms await is a slow backend — often a shared LUN, or your burst credit exhausted and IOPS throttled at the hypervisor layer.
iostat -dxm 1 5 /dev/vda | tail -20
# r/s w/s rMB/s wMB/s await r_await w_await aqu-sz %util
# who is issuing it
sudo pidstat -d 1 5
# or the slower but always-available route
for p in /proc/[0-9]*; do
echo "$(basename $p) $(awk '{s+=$6} END{print s}' $p/io 2>/dev/null)"
done | sort -k2 -n | tail -10
Watch aqu-sz vs the queue depth the device reports. On a virtio disk, cat /sys/block/vda/queue/nr_requests is the guest-side cap (often 128), but the hypervisor may only have 32 slots available for your instance. If aqu-sz exceeds roughly 25% of the reported depth while await rises linearly, you are being queued downstream and the extra in-flight requests are pure latency.
Step 4: test the floor with a controlled write
One minute of fio at a fixed queue depth separates “my workload is heavy” from “this volume cannot do what was sold”. Run it to a temp file, not the device, and compare against your recorded baseline.
sudo fio --name=floor --filename=/tmp/floor.bin --size=1G --bs=4k \
--rw=randwrite --iodepth=16 --numjobs=1 --runtime=30 --time_based \
--ioengine=libaio --direct=1 --group_reporting
# drain caches first so you measure the device, not page cache
sudo sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Read the clat percentiles, not the mean. A p99 of 12 ms with a mean of 2 ms on a 4 KiB direct write indicates the tail is being throttled, which is exactly what an IOPS credit system looks like: most requests sail through, one in a hundred queues behind the throttle window.
Step 5: check memory pressure — the confounder
Both symptoms have a third cause that mimics them: reclaim. When page cache is evicted aggressively, application reads become device reads, which shows as I/O wait; and when kswapd runs hot, runnable work looks like steal because the scheduler is busy in kernel memory paths. Before blaming the host, confirm memory is not the driver.
cat /proc/pressure/memory /proc/pressure/io
vmstat 1 3 | awk 'NR>2{print "si="$7" so="$8" free="$4" cache="$6}'
cat /proc/vmstat | grep -E "pgscan|pgsteal|allocstall"
Rising pgscan_direct with nonzero allocstall is direct reclaim stalling allocations, and that is a memory sizing issue. Fixing it — a smaller buffer pool, a tighter PHP-FPM pool, or more RAM — frequently removes both the “I/O wait” and the “steal” without any network or provider involvement.
A worked example: two incidents that looked identical
Incident A: 4 vCPU instance, load 6.1, response times tripled. vmstat showed us 22 sy 6 id 51 wa 4 st 17. PSI CPU some avg10 was 34. fio at queue depth 16 returned a healthy 0.9 ms p50, proving storage was fine. Conclusion: hypervisor CPU contention; the fix was a host migration or a dedicated-core plan, and no amount of application tuning helped.
Make the triage repeatable
Collect vmstat, mpstat -P ALL, iostat -dxm and all three PSI files every minute into a single log, and keep a week. Ninety percent of triage arguments disappear when you can show a chart of st and await over the period a user complained about. The command that usually earns its keep is the simplest: vmstat 1 60 piped to a file at the moment the alert fires, because both steal and I/O contention are bursty and a five-minute monitoring average will hide the very window you care about.
When the data points at contention rather than configuration, the decision becomes commercial: keep a shared instance with unpredictable neighbours, or move to a VPS tier with guaranteed CPU and dedicated NVMe. Either way, decide with counters in hand.




Leave a Reply
You must be logged in to post a comment.