CPU Pinning on KVM VPS Hosts: Measuring the Latency Effect

On a busy KVM host your vCPU is a thread that competes with other tenants’ threads for a physical core and for L3 cache. Pinning a guest vCPU to a dedicated host core removes the scheduler’s freedom to migrate it, which removes a large chunk of run-to-run variance. This article measures how much that actually buys, with a reproducible harness you can run on any KVM VPS in under ten minutes.

Why vCPU placement shows up as latency

The Linux CFS scheduler balances load across cores by migrating runnable threads. Each migration flushes some per-CPU state: the L1 and L2 caches go cold, and the TLB entries for the old core are no longer valid. For throughput-oriented batch work that costs a few percent. For a request-serving thread with a 2 ms budget, a migration landing mid-request can add hundreds of microseconds to the tail.

There is a second effect specific to virtualization: steal time. When the host’s runqueue is oversubscribed, your vCPU waits to be scheduled at all. Steal time is reported in /proc/stat and shows up immediately in top on the third line. Pinning helps only if the host gives you dedicated cores; on a shared plan, pinning inside the guest does nothing about host-level contention.

Confirming what the host exposes

Before measuring anything, establish the topology the guest can see. Topology information comes from the CPUID leaves the hypervisor chooses to expose, and many hosts hide it entirely — in which case guest-side pinning is impossible and the experiment ends here.

$ lscpu | sed -n '1,25p'
Architecture:            x86_64
CPU(s):                  4
On-line CPU(s) list:     0-3
Thread(s) per core:      1
Core(s) per socket:      2
Socket(s):               2
NUMA node(s):            1
Virtualization features:
  Hypervisor vendor:     KVM

$ cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
0

$ systemd-detect-virt
kvm

If thread_siblings_list shows a pair such as 0,4, the host exposed SMT and you must avoid placing two pinned vCPUs on siblings of the same physical core. lscpu -e gives the same data in tabular form and is easier to parse in a script.

Building the latency harness

The harness needs three properties: a deterministic workload, a way to isolate the measuring thread’s address space, and a record of the tail rather than just the mean. cyclictest from rt-tests is the standard tool. It sleeps for a fixed interval, measures wake-up jitter, and reports min/avg/max per thread — exactly the distribution we care about.

# apt install rt-tests stress-ng
$ cyclictest --mlockall --smp --priority=80 \
    --interval=1000 --loops=60000 --histogram=200 \
    --threads=1 -a 2 -q

T: 0 ( 4821) P:80 I:1000 C:  60000 Min:   2 Act:   7
Avg:   9 Max:  184

# tail latency as a percentage of samples above thresholds
$ cyclictest --mlockall -t1 -a2 -i1000 -D 60 -q | \
    awk '{print $0}' | grep -o 'Max: *[0-9]*'

Run the test twice per configuration: once on an idle guest, once with a competing load generator pinned elsewhere. stress-ng --cpu 1 --cpu-method matrixprod pinned to a different vCPU gives a realistic neighbour while leaving your measured thread’s core free.

Pinning with taskset and cgroups

For a quick test, taskset is enough. The masking is per-thread, so a multi-threaded process needs each thread set individually unless you use the process-wide flag before it forks.

$ taskset -c 2 cyclictest --mlockall -t1 -a0 -i1000 -D 60 -q

# check where a running process actually sits
$ taskset -cp $(pgrep -f nginx | head -1)
pid 8213's current affinity list: 2

# per-thread affinity for a multi-threaded service
$ for t in /proc/8213/task/*; do taskset -cp 2 ${t##*/}; done

For a service, express the pin as a systemd property so it survives restarts. CPUAffinity= in the unit file applies to every thread the unit spawns, and AllowedCPUs= in a slice does the same at cgroup level — the latter is preferable because it constrains children that try to set their own affinity.

# /etc/systemd/system/api.service.d/pin.conf
[Service]
CPUAffinity=2 3
CPUQuota=150%

$ systemctl daemon-reload && systemctl restart api.service
$ systemctl show api.service -p CPUAffinity -p CPUQuotaPerSecUSec

Interpreting the numbers

Across four 4-vCPU KVM instances on shared hosts, the pattern below is typical. The idle columns barely differ — pinning cannot improve a system with nothing competing. The loaded columns are where it pays: max latency dropped roughly in half, and the P99 moved by about a third. Absolute values shift with host load, so treat the ratios as the result, not the microseconds.

config              idle avg   idle max   loaded avg   loaded max   loaded p99
unpinned (CFS)         9 us      184 us       14 us       612 us       71 us
pinned, shared core   10 us      191 us       11 us       488 us       58 us
pinned, dedicated     10 us      176 us       10 us       203 us       34 us

The “pinned, dedicated” row is the only one that changed the worst case by an order of magnitude. That requires the provider to guarantee exclusivity — an arrangement you should confirm in writing, because guest-side pinning on an oversubscribed host produces the middle row at best. If steal time above 5% appears during the test, the host is contending and the measurement describes the host, not your code.

Pitfalls that invalidate the measurement

Missing --mlockall lets the measuring thread’s pages be reclaimed mid-test, producing a fake outlier spike. Running the test with the CPU governor in powersave introduces frequency transitions that look like scheduler jitter. And measuring for thirty seconds tells you nothing: wake-up latency distributions are heavy-tailed, and the interesting samples appear after several thousand iterations.

Also check whether the host exposed constant_tsc and nonstop_tsc in /proc/cpuinfo. Without them, the TSC can drift between cores and any cross-core timing becomes unreliable — a subtle failure that silently corrupts the comparison. For more on how hypervisors expose hardware, see our analysis of steal time and virtio device overhead.

Deciding whether to pin in production

Pin when a single hot thread dominates request latency and the host exposes real topology. Do not pin on a small instance where the OS, monitoring agent and application already compete for two or three vCPUs — restricting the scheduler there costs more in throughput than it saves in jitter. Measure both configurations under load, keep the raw histograms, and only adopt the pinning if the tail improves by more than the run-to-run noise you measured.

Our KVM VPS plans expose CPU topology and publish per-plan core allocation, which makes this experiment reproducible on the first try. Start with the harness above, run it before and after any placement change, and let the histogram decide.

Leave a Reply