Finding VPS Scheduler Latency with perf sched and bpftrace

“The CPU is at 30% but everything is slow” is the classic symptom of scheduler latency — the time a runnable thread waits before it actually gets a core. Aggregate CPU utilisation cannot show it, because the problem is not how busy the cores are, it is how long work sits in the run queue. On a VPS this is worse than on bare metal: your vCPUs are host threads, so their wakeup latency is partly decided by a hypervisor you do not control.

This tutorial shows how to find and quantify scheduler latency on a VPS using perf sched and bpftrace, measure it as a percentile rather than an average, and decide whether the fix is inside your guest or a host you should abandon. For related profiling methodology, see our VPS performance-tuning guides.

Why CPU utilisation lies

Load average and %idle describe throughput, not latency. A thread can be runnable and sitting in the CFS run queue while the core is technically idle in a deeper c-state, and your p99 request latency pays for every microsecond of that wait. The metric that matters is run-queue latency: time from “task becomes runnable” to “task starts executing on a CPU”.

Two tools measure it precisely: perf sched, which replays kernel scheduling events, and bpftrace, which hooks the scheduler tracepoints live at low overhead.

Step 1: get a baseline with perf sched

perf sched record captures the scheduling events and perf sched latency summarises them per task. Run it during your slow period, not at idle.

• perf sched record -- sleep 30 — capture 30 seconds of scheduling activity.
• perf sched latency — the summary table. Sort mentally on the Maximum and Average columns.
• Look for the columns: task, runtime, swait count, then wait time and sch delay. High wait time on your worker threads is the smoking gun.

Interpretation is straightforward: wait time is how long the task was runnable but off-CPU, and sch delay is how long it waited to be scheduled after waking. If a request-handling thread shows average wait times in the hundreds of microseconds during your slow window, the scheduler — not your code — is adding that latency to every request.

What you see	Likely cause	Where to look
High wait on all threads, steal also high	Host oversubscription	mpstat %steal
High wait on a few threads	They share a core with a CPU hog	cgroup CPU weight
High sch delay on wakeups	Many wakeups, low nr_running skew	Wakeup coalescing
Spikes at fixed intervals	Timer or cron contention	systemd-analyze, cron

Step 2: live histograms with bpftrace

perf sched gives you a post-hoc summary. To watch run-queue latency as a live distribution, hook the scheduler tracepoints with a one-line bpftrace script. This gives you percentiles without storing an entire trace.

Measure time-on-CPU between context switches as a rough proxy for queueing, aggregated into a histogram:

• bpftrace -e 'tracepoint:sched:sched_switch { @run[tid] = nsecs; }
• tracepoint:sched:sched_wakeup /@run[args->pid]/ { @lat = hist(nsecs - @run[args->pid]); delete(@run[args->pid]); }'

That prints a log2 histogram of wakeup-to-running latency. Read the tail: if the histogram has significant mass beyond, say, 128 µs for an interactive workload, the scheduler is adding perceivable delay. A healthy, unloaded guest keeps nearly all wakeups under a few tens of microseconds.

Step 3: separate guest latency from host latency

This is the step most people skip, and it decides whether you can fix the problem at all. Inside a KVM guest, part of your scheduling wait is the host refusing to run your vCPU thread.

• Run mpstat -P ALL 1 60 while your bpftrace script runs. If wakeup latency spikes correlate with %steal spikes, the latency originates outside your guest.
• Check cat /proc/pressure/cpu (PSI). A rising some/full avg10 confirms genuine CPU contention rather than a single slow thread.
• If steal is near zero and PSI is low but latency is still high, the problem is inside the guest: too many runnable threads for the vCPU count, or a priority inversion.

Fixing scheduler latency you can control

• Reduce runnable threads. Match worker/process counts to vCPU count; over-provisioning worker pools raises run-queue depth and inflates every task’s wait time.
• Use cgroup v2 CPU weights. Give latency-sensitive services a higher cpu.weight and lower-weighted batch jobs, so they get scheduled first without starving anyone.
• Consider a real-time or low-latency profile. chrt -f for a genuinely latency-critical thread, or a SCHED_BATCH hint for background work.
• Pin hot threads. taskset a busy worker to a dedicated vCPU so it stops fighting the packer for a core.
• Cut timer wakeups. Coalescing timers in the application reduces wakeup storms that drive sch delay up across all threads.

Verify the fix with the same tools

Re-run the exact bpftrace histogram and perf sched latency under identical load. Success looks like: the histogram’s 99th-percentile bucket moving down by a factor, perf sched‘s maximum wait dropping, and request p99 latency falling by a similar amount. If guest-side changes do nothing and the bpftrace histograms still track host steal time, no amount of in-guest tuning will help — that is the signal to move the workload to a less oversubscribed instance, using a VPS provider comparison measured on steal time and latency, and benchmark again.