{"id":1372,"date":"2026-10-12T23:13:14","date_gmt":"2026-10-12T23:13:14","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1372"},"modified":"2026-10-12T23:13:14","modified_gmt":"2026-10-12T23:13:14","slug":"finding-vps-scheduler-latency-perf-bpftrace","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/","title":{"rendered":"Finding VPS Scheduler Latency with perf sched and bpftrace"},"content":{"rendered":"<div style=\"padding:20px\">\n\n<p class=\"wp-block-paragraph\">&#8220;The CPU is at 30% but everything is slow&#8221; is the classic symptom of scheduler latency &mdash; the time a runnable thread waits before it actually gets a core. Aggregate CPU utilisation cannot show it, because the problem is not how busy the cores are, it is how long work sits in the run queue. On a VPS this is worse than on bare metal: your vCPUs are host threads, so their wakeup latency is partly decided by a hypervisor you do not control.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This tutorial shows how to find and quantify scheduler latency on a VPS using <code>perf sched<\/code> and <code>bpftrace<\/code>, measure it as a percentile rather than an average, and decide whether the fix is inside your guest or a host you should abandon. For related profiling methodology, see our <a href=\"https:\/\/virtualserversvps.com\/\">VPS performance-tuning guides<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why CPU utilisation lies<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Load average and <code>%idle<\/code> describe throughput, not latency. A thread can be runnable and sitting in the CFS run queue while the core is technically idle in a deeper c-state, and your p99 request latency pays for every microsecond of that wait. The metric that matters is <strong>run-queue latency<\/strong>: time from &#8220;task becomes runnable&#8221; to &#8220;task starts executing on a CPU&#8221;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Two tools measure it precisely: <code>perf sched<\/code>, which replays kernel scheduling events, and <code>bpftrace<\/code>, which hooks the scheduler tracepoints live at low overhead.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1: get a baseline with perf sched<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><code>perf sched record<\/code> captures the scheduling events and <code>perf sched latency<\/code> summarises them per task. Run it during your slow period, not at idle.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>perf sched record -- sleep 30<\/code> &mdash; capture 30 seconds of scheduling activity.<\/li>\n<li><code>perf sched latency<\/code> &mdash; the summary table. Sort mentally on the <code>Maximum<\/code> and <code>Average<\/code> columns.<\/li>\n<li>Look for the columns: task, runtime, swait count, then <code>wait time<\/code> and <code>sch delay<\/code>. High <code>wait time<\/code> on your worker threads is the smoking gun.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Interpretation is straightforward: <code>wait time<\/code> is how long the task was runnable but off-CPU, and <code>sch delay<\/code> is how long it waited to be scheduled after waking. If a request-handling thread shows average wait times in the hundreds of microseconds during your slow window, the scheduler &mdash; not your code &mdash; is adding that latency to every request.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>What you see<\/th><th>Likely cause<\/th><th>Where to look<\/th><\/tr><\/thead><tbody><tr><td>High wait on all threads, steal also high<\/td><td>Host oversubscription<\/td><td><code>mpstat %steal<\/code><\/td><\/tr><tr><td>High wait on a few threads<\/td><td>They share a core with a CPU hog<\/td><td>cgroup CPU weight<\/td><\/tr><tr><td>High sch delay on wakeups<\/td><td>Many wakeups, low nr_running skew<\/td><td>Wakeup coalescing<\/td><\/tr><tr><td>Spikes at fixed intervals<\/td><td>Timer or cron contention<\/td><td><code>systemd-analyze<\/code>, cron<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2: live histograms with bpftrace<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><code>perf sched<\/code> gives you a post-hoc summary. To watch run-queue latency as a live distribution, hook the scheduler tracepoints with a one-line bpftrace script. This gives you percentiles without storing an entire trace.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Measure time-on-CPU between context switches as a rough proxy for queueing, aggregated into a histogram:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>bpftrace -e 'tracepoint:sched:sched_switch { @run[tid] = nsecs; }<\/code><\/li>\n<li><code>tracepoint:sched:sched_wakeup \/@run[args-&gt;pid]\/ { @lat = hist(nsecs - @run[args-&gt;pid]); delete(@run[args-&gt;pid]); }'<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">That prints a log2 histogram of wakeup-to-running latency. Read the tail: if the histogram has significant mass beyond, say, 128&nbsp;&micro;s for an interactive workload, the scheduler is adding perceivable delay. A healthy, unloaded guest keeps nearly all wakeups under a few tens of microseconds.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3: separate guest latency from host latency<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This is the step most people skip, and it decides whether you can fix the problem at all. Inside a KVM guest, part of your scheduling wait is the host refusing to run your vCPU thread.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Run <code>mpstat -P ALL 1 60<\/code> <em>while<\/em> your bpftrace script runs. If wakeup latency spikes correlate with <code>%steal<\/code> spikes, the latency originates outside your guest.<\/li>\n<li>Check <code>cat \/proc\/pressure\/cpu<\/code> (PSI). A rising <code>some<\/code>\/<code>full avg10<\/code> confirms genuine CPU contention rather than a single slow thread.<\/li>\n<li>If steal is near zero and PSI is low but latency is still high, the problem is inside the guest: too many runnable threads for the vCPU count, or a priority inversion.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Fixing scheduler latency you can control<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Reduce runnable threads.<\/strong> Match worker\/process counts to vCPU count; over-provisioning worker pools raises run-queue depth and inflates every task&#8217;s wait time.<\/li>\n<li><strong>Use cgroup v2 CPU weights.<\/strong> Give latency-sensitive services a higher <code>cpu.weight<\/code> and lower-weighted batch jobs, so they get scheduled first without starving anyone.<\/li>\n<li><strong>Consider a real-time or low-latency profile.<\/strong> <code>chrt -f<\/code> for a genuinely latency-critical thread, or a <code>SCHED_BATCH<\/code> hint for background work.<\/li>\n<li><strong>Pin hot threads.<\/strong> <code>taskset<\/code> a busy worker to a dedicated vCPU so it stops fighting the packer for a core.<\/li>\n<li><strong>Cut timer wakeups.<\/strong> Coalescing timers in the application reduces wakeup storms that drive sch delay up across all threads.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Verify the fix with the same tools<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Re-run the exact bpftrace histogram and <code>perf sched latency<\/code> under identical load. Success looks like: the histogram&#8217;s 99th-percentile bucket moving down by a factor, <code>perf sched<\/code>&#8216;s maximum wait dropping, and request p99 latency falling by a similar amount. If guest-side changes do nothing and the bpftrace histograms still track host steal time, no amount of in-guest tuning will help &mdash; that is the signal to move the workload to a less oversubscribed instance, using a <a href=\"https:\/\/virtualserversvps.com\/\">VPS provider comparison measured on steal time and latency<\/a>, and benchmark again.<\/p>\n\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;The CPU is at 30% but everything is slow&#8221; is the classic symptom of scheduler latency &mdash; the time a runnable thread waits before it actually gets a core. Aggregate&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":1,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1372","post","type-post","status-publish","format-standard","hentry","category-vps-guides-tutorials"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Finding VPS Scheduler Latency with perf sched and bpftrace - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"&quot;The CPU is at 30% but everything is slow&quot; is the classic symptom of scheduler latency &mdash; the time a runnable thread waits before it actually gets a\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Finding VPS Scheduler Latency with perf sched and bpftrace\" \/>\n<meta property=\"og:description\" content=\"Finding VPS Scheduler Latency with perf sched and bpftrace\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-12T23:13:14+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/\",\"name\":\"Finding VPS Scheduler Latency with perf sched and bpftrace - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-10-12T23:13:14+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"\\\"The CPU is at 30% but everything is slow\\\" is the classic symptom of scheduler latency &mdash; the time a runnable thread waits before it actually gets a\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Finding VPS Scheduler Latency with perf sched and bpftrace\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Finding VPS Scheduler Latency with perf sched and bpftrace - Virtual Servers VPS Blog","description":"\"The CPU is at 30% but everything is slow\" is the classic symptom of scheduler latency &mdash; the time a runnable thread waits before it actually gets a","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/","og_locale":"en_US","og_type":"article","og_title":"Finding VPS Scheduler Latency with perf sched and bpftrace","og_description":"Finding VPS Scheduler Latency with perf sched and bpftrace","og_url":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-10-12T23:13:14+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/","url":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/","name":"Finding VPS Scheduler Latency with perf sched and bpftrace - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-10-12T23:13:14+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"\"The CPU is at 30% but everything is slow\" is the classic symptom of scheduler latency &mdash; the time a runnable thread waits before it actually gets a","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/finding-vps-scheduler-latency-perf-bpftrace\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Finding VPS Scheduler Latency with perf sched and bpftrace"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1372","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1372"}],"version-history":[{"count":2,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1372\/revisions"}],"predecessor-version":[{"id":1378,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1372\/revisions\/1378"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1372"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1372"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1372"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}