Reading Linux Performance in Under 60 Seconds: vmstat, pidstat and PSI

A slow server does not hide its bottleneck: the Linux kernel accounts for every nanosecond of CPU time, every major page fault, every block-layer request and every dropped packet. The difficulty is turning that firehose into a decision in under a minute. This is a drill-down procedure — one-second system snapshot, then per-process attribution, then device-level and memory-level confirmation — using only tools that ship with a normal distribution or one apt install sysstat. All of it reads from /proc and /sys, so it works identically on any Linux instance, whether that is a properly sized VPS with dedicated CPU shares or a cramped container.

Step 1: Establish the Baseline Before Anything Is Wrong

Tuning and diagnosis both fail without a baseline. Capture one while the box is healthy, so a later reading has something to be compared against:

<code>mkdir -p /var/log/baseline
vmstat 1 60 > /var/log/baseline/vmstat.txt
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io \
  > /var/log/baseline/psi.txt
sar -u -r -d 1 5 > /var/log/baseline/sar.txt</code>

Two numbers matter most in that baseline: r (run queue) and the PSI avg10 values. A healthy idle box shows r at or below the vCPU count and PSI avg10 under 1.00 for all three resources.

Step 2: The One-Second Snapshot

vmstat 1 remains the densest single command on Linux. Read the columns in a fixed order rather than scanning them:

<code>vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st</code>

The st row deserves emphasis. Steal time means your vCPU was runnable but the hypervisor scheduled someone else. If the baseline was 0.2% and the current reading is 6%, the instance is being oversubscribed on the host, and every sysctl in the following sections is theatre. Confirm with mpstat -P ALL 1 to see whether steal is spread across all vCPUs or concentrated on one — concentrated steal usually indicates a co-scheduled noisy neighbour or a pinned host thread.

Step 3: Attribute CPU to a Process

top averages over its refresh interval and hides short spikes. pidstat reports true deltas between samples and separates the accounting:

<code>sudo apt install sysstat
pidstat -u -p ALL 1      # user/system CPU per process
pidstat -r -p ALL 1      # page faults, RSS growth
pidstat -d -p ALL 1      # disk read/write per process
pidstat -w -p ALL 1      # voluntary/involuntary context switches</code>

Read %usr and %system separately. A process at 100% %system with a small %usr is spending its time in syscalls — typically small-buffer read/write loops, excessive futex traffic, or a tracing agent. That is tunable or starvable; a process at 100% %usr needs a faster core or fewer threads.

pidstat -w is the underrated one. Involuntary context switches (nvcswch/s) climbing on a process whose %CPU is below 100 means it is being preempted — either the run queue is full or its CPU affinity is competing with another hot thread. Voluntary switches (cswch/s) in the tens of thousands per second usually mean lock contention, not I/O, especially in JVM and Go workloads.

Step 4: Localise Disk Pressure Properly

High wa in vmstat points at storage; iostat -x 1 tells you which layer. The trap is that %util is nearly meaningless on NVMe, where a device can service hundreds of parallel requests and still show high utilisation at trivial latency.

<code>iostat -x 1
# Device  r/s   w/s  rkB/s  wkB/s  r_await  w_await  aqu-sz  %util</code>

• r_await/w_await are the numbers that matter. On local NVMe, sub-millisecond is normal; on network-backed block storage, 0.5–2 ms is expected and fine.
• aqu-sz rising while latency stays flat means the queue is absorbing bursts — healthy. Rising aqu-sz with rising latency means the device is the wall.
• Compare guest w/s against the provider’s throttle. Many plans cap IOPS; when the cap is hit, latency jumps in a step, not a slope. A step change at a suspiciously round number is the signature.

Cross-check with PSI, which quantifies the time someone was actually stalled rather than how busy the device looked:

<code>cat /proc/pressure/io
# some avg10=12.43 avg60=8.11 avg300=3.02 total=9183421
# full avg10=9.87 avg60=6.44 avg300=2.11 total=7211233</code>

The full line means every runnable task was stalled on I/O. Sustained full avg10 above 5 on a database host is a capacity problem, not a tuning problem.

Step 5: Memory — Beyond “Free RAM”

free -m is the most misread command in the toolbox: low free with a large buff/cache is correct kernel behaviour. What matters is reclaim pressure.

<code>grep -E 'pgscan|pgsteal|allocstall|oom' /proc/vmstat
cat /proc/pressure/memory
grep -E 'AnonHugePages|Shmem|FilePmdMapped' /proc/meminfo
slabtop -o -s c | head -15</code>

Rising pgscan_direct and allocstall mean allocations are blocking on reclaim — the working set no longer fits. A large slabtop dentry or inode cache after a file-heavy workload is normal; a runaway kmalloc-* slab is a kernel leak, usually from a filesystem or network driver, and shows up as steady growth that survives a workload restart.

Then confirm with per-process RSS versus PSS, because RSS double-counts shared pages:

<code>sudo smem -t -k -c "pname uss pss rss" | sort -k3 -h | tail -10</code>

Step 6: The 60-Second Decision

1. vmstat 1 1 — if st is high, stop; the host is the problem.
2. If si/so are non-zero, go to memory and fix the working set or add RAM.
3. If r > vCPUs with low wa, go to pidstat -u and find the hog or the thread explosion.
4. If wa is high, go to iostat -x, compare latency against the baseline, and check PSI io full.
5. If all three are clean but the application is slow, the bottleneck has moved outside the kernel: DNS, TLS handshakes, an upstream dependency, or the client’s network.

Instrument the result. A five-line cron job that appends the five decisive numbers every minute gives you the history needed to correlate incidents:

<code>*/1 * * * * /usr/bin/vmstat 1 1 | tail -1 >> /var/log/baseline/minute.log</code>

Rotate it with logrotate at a 7-day retention — roughly 3 MB at that rate — and you have a forensic trail that answers “was it us or the host” without a monitoring stack. If the readings consistently show the instance is the smaller half of the equation, sizing is the fix rather than more tuning: see the provider plan dimensions and CPU allocation model before spending another evening chasing sysctl values. Measurement first, provisioning second, tuning third — in that order, the numbers always win.