Diagnosing High Load Average on a VPS: Runnable vs Blocked Task Triage

A load average of 12 on a 4 vCPU box should mean everything is crawling. Sometimes it does, and sometimes the box serves requests in 40 ms while uptime reports numbers that look like a denial of service. The difference is whether the processes counted as “runnable” are actually consuming CPU or are blocked in uninterruptible sleep waiting for a disk that never quite keeps up.

Linux load average counts two things: tasks in the run queue (TASK_RUNNING, which includes running and runnable) and tasks in TASK_UNINTERRUPTIBLE sleep — the D state. That second category is why load average diverges from CPU utilisation, and it is why the first diagnostic step is not “add CPU” but “find out which of these two categories is inflating the number.”

Split the load into runnable and blocked

/proc/loadavg gives the total, and the two numbers after the running-task count tell you the split directly. The fourth field is running/total, where “running” is the run-queue count and “total” is every process including blocked ones.

cat /proc/loadavg
# 12.44 10.81 9.02 2/412 88213
#                               ^ running / total tasks
# value 2 = two tasks in R state, so ~10 of the load is blocked I/O

# task-state breakdown, the authoritative view
ps -eo state,pid,comm --no-headers | awk '{c[$1]++} END {for (s in c) printf "%s=%d ", s, c[s]; print ""}'
ps -eo state,pid,stat,wchan:24,comm | awk '$1 ~ /^D/' | head -20

The wchan column names the kernel function the blocked task is sleeping in. On a healthy box it is empty or shows short-lived waits; on a struggling VPS it shows something like io_schedule, ext4_writepages, blk_mq_get_tag, or rq_qos_throttle. Those last two are the signature of a storage device that has hit its IOPS or queue-depth limit — either your provider’s throttle, or the queue depth of a single virtio device.

The three-way test: CPU, memory, or disk

Run these three commands in parallel for 30 seconds. The pattern of which one shows pressure tells you which subsystem to investigate; running only one of them is how people end up buying more RAM for a disk problem.

# 1. CPU: is the run queue real work?
vmstat 2 15 | awk 'NR==1 || /^[0-9]/ {print $1,$2,$13,$14,$15,$16,$17}'
# columns: r=runqueue, us, sy, id, wa, st

# 2. memory: is anything swapping?
sar -r 2 15 2>/dev/null || vmstat -s | egrep 'swapped|free'

# 3. disk: is the device the bottleneck?
iostat -x 2 15 /dev/vda

The r column from vmstat is the run queue excluding blocked tasks. If r is 2 and load is 12, ten tasks are blocked and the CPU is not the problem. If r is 14 on a 4 vCPU box and us+sy is above 90%, then it is genuinely CPU-bound and the next question is whether that CPU is yours or the host’s — check st (steal).

Pattern	vmstat r	wa %	st %	Cause
CPU saturated by your work	≥ nproc	< 5	< 2	Genuine CPU load
Host contention	≥ nproc	< 5	> 5	Noisy neighbour
Disk stall	low	> 20	< 2	I/O queue full
Memory pressure	low–mid	> 10	< 2	Swap thrash
Provider throttle	low	> 30	< 2	IOPS cap

Case 1: it is your CPU

When r is at or above the vCPU count and %idle approaches zero, find the consumer before assuming you need a bigger plan. Top’s default view hides the real distribution because it shows instantaneous CPU percentage; pidstat accumulated over a window is more reliable.

pidstat -u 5 6 | sort -k8 -nr | head -15
# per-thread view, catches single-threaded hogs averaged out by top
ps -eLf --sort=-pcpu | head -20
# is one process running on one core while others idle?
mpstat -P ALL 2 5

If mpstat shows one core at 100% and the rest near idle while load reports 12, the load number is coming from blocked tasks, not the run queue — go to case 3. If all cores are busy, the workload genuinely needs more vCPU or a configuration change: PHP-FPM pm.max_children set far too high, a MySQL query without an index doing filesort across a large table, or a backup process running at the wrong hour.

Case 2: CPU steal is inflating the run queue

High st with your own processes runnable is the noisy-neighbour signature. Your tasks are counted as runnable because they could execute, but the host is not scheduling the guest’s vCPU. This produces high load average and high latency without any process in your guest consuming unusual CPU.

# steal over a 30s window
python3 - <<'PY'
import time
def snap():
    f=open('/proc/stat').readline().split()[1:]
    return [int(x) for x in f]
a=snap(); time.sleep(30); b=snap()
d=[y-x for x,y in zip(a,b)]; t=sum(d)
print(f"user {100*d[0]/t:.1f}% sys {100*d[2]/t:.1f}% idle {100*d[3]/t:.1f}% "
      f"iowait {100*d[4]/t:.1f}% steal {100*d[7]/t:.1f}%")
PY

Sustained steal above 5% on a paid plan is worth a support ticket with the numbers attached. Above 10% for hours, the instance is on a host that is materially oversubscribed, and no in-guest tuning addresses it. Sustained steal is also the single best predictor of whether a plan you are trialling will perform, which is why our VPS performance guide puts it first in the trial-hour checklist.

Case 3: the disk is the run queue

This is the most common high-load-on-a-VPS cause and the least intuitive. A device with a queue depth of 128 that is servicing requests at 400 IOPS will hold 128 requests in flight and add more to the backlog, each one counted in load average. The CPU sits at 30% idle, wa is above 30, and every request takes four seconds.

iostat -x 2 10
# watch: aqu-sz (queue size), await (ms), %util
# await > 50ms with aqu-sz > 4 on NVMe = saturated device

# which processes are generating the I/O?
iotop -oPa -d 3 -n 3
# per-process io accounting from the kernel
cat /proc/$(pgrep -f mysql | head -1)/io

await above 20 ms on a device that reported sub-100 µs latency at idle means you are queueing. Distinguish transient from sustained: a checkpoint flush every five minutes is expected; a permanent queue means the workload exceeds the plan’s sustained IOPS. The usual in-guest remedies are reducing write amplification (log rotation intervals, innodb_flush_log_at_trx_commit, journal commit interval), moving cold data off the volume, and checking that no backup or package upgrade is overlapping with peak traffic.

# ext4: raise the journal commit interval from 5s default for write-heavy loads
mount -o remount,commit=30 /
# verify
findmnt -no OPTIONS /

Note the durability trade: a 30-second commit interval means up to 30 seconds of writes can be lost on a hard host failure. For a rebuildable cache or a log volume, that is fine. For a billing database, it is not. Measure before you change it — if await is already low, the journal interval is not your bottleneck.

Case 4: memory pressure masquerading as load

When the kernel reclaims pages faster than applications fault them back in, tasks block in direct reclaim and the load average climbs with no CPU or disk utilisation to show for it. The tell is swap activity alongside a page cache that has collapsed.

vmstat 2 10 | awk 'NR==1 || /^[0-9]/ {print $1,$3,$4,$7,$8}'
# si/so non-zero and sustained = swapping

awk '/MemAvailable|Cached|SwapTotal|SwapFree|Committed_AS/ {print}' /proc/meminfo
# identify what is holding RSS
ps_mem -s 2>/dev/null | head -15 || ps -eo rss,comm --sort=-rss | head -15

A useful threshold: if MemAvailable is under 10% of total and si/so show hundreds of KB/s sustained, the working set does not fit. That is a sizing decision, not a tuning one, though vm.swappiness and per-service memory limits can buy time. The workload sizing guide linked from our main site maps RAM requirements by service type if you are deciding whether to resize.

Tools that identify blocked tasks precisely

For a single event rather than a trend, capture what is blocked at the moment of the spike. /proc/PID/stack gives the kernel stack of a blocked task (root only), and perf or bpftrace can sample off-CPU time to attribute waiting to a specific call path.

# kernel stack of each D-state task
for p in $(ps -eo pid,state --no-headers | awk '$2 ~ /^D/{print $1}'); do
  echo "=== $p $(cat /proc/$p/comm)"; cat /proc/$p/stack 2>/dev/null | head -6
done

# off-CPU time by process, 10s sample (needs bcc-tools)
offcputime-bpfcc -d 10 2>/dev/null | head -25

# syscall latency outliers for a specific process
strace -c -f -p $(pgrep -f nginx | head -1) & sleep 10; kill %1

The offcputime output is the one that settles arguments. If 90% of blocked time lands in io_schedule under the database’s threads, the disk is the answer regardless of what top suggests. If it lands in futex waits, the bottleneck is internal locking in your application and neither CPU nor disk will help.

A one-page triage order

1. cat /proc/loadavg — split runnable from total. If runnable is low, the load is blocked I/O or memory.
2. ps -eo stat,wchan,comm | grep ^D — name the kernel function being waited on.
3. vmstat 2 15 — read r, wa, st, si/so together.
4. iostat -x 2 10 — await and aqu-sz confirm or rule out the device.
5. pidstat -u 5 6 — attribute CPU only after the first four rules out I/O and memory.

Ninety percent of “the server is overloaded” reports resolve at step two or three. The load average is an aggregate that answers no question on its own; the split and the wchan are what convert it into a diagnosis.