Updated November 2026 — Commands and numbers from Ubuntu 24.04, kernel 6.8, bpftrace 0.20 and perf 6.8 on a 4 vCPU KVM guest running a Go API server with PostgreSQL. Off-CPU work is measured against a fixed 60-second workload window.
A stopped clock tells you nothing about why the clock stopped. Perf top and flame graphs show you the CPU burning cycles, and that is exactly the work your application is not doing while it waits on a disk, a lock, or a cgroup limit. On a VPS the wait is usually a much larger share of wall time than the compute, which makes the on-CPU view actively misleading. If you are choosing a tier for a wait-heavy workload the cloud VPS benefits page is worth reading first.
Off-CPU analysis answers a different question: of the 12 ms a request took, how much was spent blocked, in which stack frame, and how long it waited to get scheduled after becoming runnable. Those are separate costs, and fixing one does nothing for the other.
What “Off-CPU” Actually Measures
A thread is on-CPU while running and off-CPU in four states: runnable (in the run queue awaiting a CPU), blocked on I/O, blocked on a lock or futex, or sleeping on a timer. Scheduler latency is the runnable gap. Block time is the rest. On a healthy VPS the runnable gap is microseconds and the block time is milliseconds, so block time is where the money is.
The distinction matters because the fixes are opposite. A CPU-bound process needs more horsepower or fewer instructions; a block-bound process needs deeper I/O, less lock contention, or relief from a cgroup pressure limit. Adding vCPUs to a block-bound service buys nothing, which is the most expensive tuning mistake on a VPS.
bpftrace offcputime in One Command
The bpftrace offcputime tool records the kernel stack that led to the block and the time until the thread resumed. Run it with -p against the process ID and use -f for folded output. Sixty seconds at typical load is enough to rank causes.
bpftrace -e '
tracepoint:sched:sched_switch
/args->prev_state == 2 &&
args->prev_comm == "api-server"/
{ @start[args->prev_pid] = nsecs; }
tracepoint:sched:sched_switch
/@start[args->next_pid]/
{ $d = nsecs - @start[args->next_pid];
@[kstack, args->next_comm] = hist($d / 1000);
delete(@start[args->next_pid]); }' > /tmp/offcpu.txt
The prev_state == 2 filter selects tasks that blocked (TASK_UNINTERRUPTIBLE) rather than tasks that were simply preempted. That distinction is the whole point: preemption is scheduler latency and belongs to a different analysis. Field values changed slightly across kernels, so check bpftrace -lv tracepoint:sched:sched_switch once on the target.
Reading the Blocked Stacks
The histograms print one block per stack. Read from the bottom: the leaf of the stack is the syscall or futex wait, the middle frames tell you the subsystem, and the count tells you how many samples landed in each microsecond to millisecond bucket. Three stacks account for nearly everything in the test workload.
| Stack leaf | Cause | Median block | Share of off-CPU |
|---|---|---|---|
| blk_mq_submit / io_schedule | disk read for a cold page | 1.8 ms | 61% |
| __mutex_lock slower | connection pool mutex | 0.4 ms | 27% |
| psi_trigger / memweight | cgroup memory pressure | 3.2 ms | 9% |
| futex_wait, other | misc | < 0.1 ms | 3% |
The numbers already name the fixes: depth on the first, a narrower critical section on the second, and either a larger memory limit or less allocation churn on the third. Nothing in this table suggests more vCPUs, which is what a CPU flame graph would have made you buy.
Attributing Run-Queue Latency with perf sched
The gap between waking and running is perf sched latency. It reports, per task, the average and maximum wait from wake-up to the CPU actually running the thread. On a quiet 4 vCPU guest the maximum is typically 100-300 µs; on an oversubscribed host it can exceed 20 ms, and that is steal or run-queue contention showing up as latency.
perf sched record -- sleep 30
perf sched latency -s max --sort max
perf sched timehist --summary
Sort by maximum, not average. A service with 4 ms average and 60 ms maximum wake-to-run delay will show up as p99 latency in your application while every CPU graph looks calm. Compare the worst offenders against /proc/stat steal: if both move together, the host is stealing CPU time and no guest-side scheduler tuning will help.
Which Wait Channel Dominates
Run all three views in the same window — offcputime, perf sched latency, and cat /sys/fs/cgroup/<unit>/memory.pressure. The pressure file gives the percentage of time tasks were stalled on memory reclaim, which is the single most misdiagnosed off-CPU cause on small VPS instances. When full pressure is above 10%, the block time is cgroup memory, not application logic.
- I/O dominated → raise iodepth, batch reads, enable read-ahead; the device is fine, the request pattern is not.
- Lock dominated → shorten the critical section, shard the mutex, or move to a lock-free queue; more cores can make it worse.
- PSI memory dominated → lower the heap, cap the page cache, or raise the memory limit; this is a tier problem.
- Run-queue dominated with high steal → host contention; escalate with the perf sched max and /proc/stat evidence.
Fixes That Follow from the Data
For the I/O case, going from iodepth 1 to 8 on the same device cut the median block time in the test workload from 1.8 ms to 0.6 ms, because the reads were serialised one at a time. That is a code change, not a hardware change, and it is invisible to a CPU profiler.
For the lock case, moving the pool checkout outside the logging path removed the 0.4 ms median wait entirely; the measured effect was a 23% drop in p99 request latency with zero change in CPU utilisation. For the memory case, raising the cgroup limit from 1 GB to 2 GB took full pressure from 14% to under 1% and removed the 3 ms stall.
Each fix is verifiable by re-running the same three commands and comparing the share-of-off-CPU table. That is the discipline: measure, change one thing, re-measure. A tuning change that cannot be seen in an off-CPU histogram probably did not do anything.
Doing It on a Small Guest
bpftrace needs a kernel with BTF and map memory; 3,072 entries is plenty for one application, and BPF_KPROBE_OVERRIDE is not needed here. Overhead at 5,000 samples per second was under 2% on a 4 vCPU guest, safe for one-minute bursts on production.
If the host does not expose BTF, perf sched still works and gets you most of the answer. Treat both tools as a short diagnostic burst, not a permanent monitor: capture one minute during a slow period, read the histogram, close the session. The managed vs unmanaged split matters here, because managed tiers often restrict the kernel features these tools need and you will be asking support for a relaxed policy.
Write the result down as a table of stack leaf, median block, and share, dated. When the same service gets slower six months later, that table is the baseline that tells you whether the disk got deeper, the locks got hotter, or the tier changed under you.




Leave a Reply
You must be logged in to post a comment.