io_uring SQPOLL on a KVM Guest: When the Polling Thread Beats Interrupts, and When It Burns a Core

At 4 KiB random reads on a 4 vCPU KVM guest with 4 virtio-blk queues, io_uring without SQPOLL tops out near 92k IOPS while --sqthread_poll=1 reaches about 118k – a 28% gain that costs one permanently saturated vCPU. At 1 vCPU, the same flag cuts throughput by roughly 15%, because the polling thread now competes with the application for the only core available. SQPOLL is not a general upgrade; it is a trade of a dedicated core for the elimination of one syscall per batch, and whether that trade wins depends on three numbers: vCPU count, queue depth, and whether the kernel thread can be kept awake.

What SQPOLL Actually Removes

Without SQPOLL, every submission batch costs a transition into the kernel – io_uring_enter – which on a KVM guest also means a VM exit and re-entry if the kernel return path crosses a hypercall boundary. With IORING_SETUP_SQPOLL, the kernel spawns a thread that spins on the submission queue; the application writes SQEs and updates the tail pointer, and never enters the kernel unless it needs to wait for completions. The saving per operation is a few hundred nanoseconds at best, so the mechanism only matters when the syscall cost is a meaningful fraction of total work – typically at high queue depths with fast devices.

# Userspace sees SQPOLL as a flag at ring setup
io_uring_queue_init(4096, &ring,
    IORING_SETUP_SQPOLL | IORING_SETUP_SQ_AFF);
# SQ_AFF: pin the sq thread to sq_thread_cpu
# Requires CAP_SYS_NICE (or root) - otherwise setup fails with EPERM

# Check the thread exists and where it runs
ps -eLo pid,tid,psr,pcpu,comm | grep iou-sqp
# One thread per ring, named iou-sqp-<pid>
chrt -p $(pgrep iou-sqp | head -1)   # SCHED_IDLE by default

Two failure modes surface immediately in testing. Unprivileged processes cannot set up SQPOLL without IORING_SETUP_R_DISABLED and later enablement, or the kernel’s io_uring_disabled sysctl permitting it – on many distributions kernel.io_uring_disabled=2 disables io_uring entirely and every SQPOLL attempt returns -EPERM. And without SQ_AFF, the polling thread wanders across vCPUs, so the CPU cost lands wherever the scheduler puts it rather than on the core you budgeted.

The Enter-Idle Path Is What Actually Costs You

A spinning kernel thread does not spin forever. When the submission queue stays empty for sq_thread_idle milliseconds (default 1000), the thread calls schedule() and wakes only on the next submit, which triggers a syscall again. Under a steady workload this never happens; under a bursty one, every burst boundary costs a wakeup plus the mode switch SQPOLL was supposed to avoid. If your workload alternates between 50 ms of I/O and 200 ms of computation, SQPOLL can measure slower than plain io_uring.

# Tighten idle timeout so the thread stays hot through short gaps
io_uring_params p = {0};
p.flags = IORING_SETUP_SQPOLL | IORING_SETUP_SQ_AFF;
p.sq_thread_idle = 5000;    # ms, must be >= 1000
p.sq_thread_cpu = 3;        # pin to cpu3

# Kernel-side watchdog: threads that spin and are not woken
grep -i 'io_uring' /proc/sys/kernel/io_uring_disabled

# Scheduler view of the pinned thread
perf top -C 3 --no-children

The idle timeout also has a security dimension that distros care about: a spinning thread retains elevated credentials and blocks RLIMIT_NPROC accounting, which is why Ubuntu and others ship kernel.io_uring_disabled defaults that restrict unprivileged use. On a VPS you control, setting it is your call; in a multi-tenant container it usually is not.

Measuring Both Modes With fio

fio’s io_uring engine accepts the polling flags directly, which makes an apples-to-apples comparison straightforward. Run both against the same file, same runtime, same depth, and watch the IOPS line plus usr/sys CPU percentages – throughput without CPU context is the classic misreading of this benchmark.

# Interrupt-driven baseline
fio --name=base --filename=/mnt/test/f --size=8G --rw=randread \
    --bs=4k --iodepth=64 --numjobs=4 --ioengine=io_uring \
    --direct=1 --runtime=60 --time_based --group_reporting

# SQPOLL, one poll thread
fio --name=sqpoll --filename=/mnt/test/f --size=8G --rw=randread \
    --bs=4k --iodepth=64 --numjobs=4 --ioengine=io_uring \
    --sqthread_poll=1 --fixedbufs=1 \
    --direct=1 --runtime=60 --time_based --group_reporting

# Watch where the CPU goes during the run
pidstat -u -p ALL 1 60 | grep -E 'iou-sqp|fio|cpu'

Read three numbers from each report: IOPS, sys time, and p99 latency. SQPOLL should reduce sys sharply – that is the mechanism working. If sys stays flat and IOPS rises, the gain came from something else and is probably noise or page-cache behaviour, since --direct=1 was supposed to remove that variable. Run with the file pre-filled (--fill_device or a prior write pass) so you are not measuring allocation.

Kernel and Queue-Depth Caveats

SQPOLL is only worth enabling above a threshold. Below roughly iodepth 8 per ring, the syscall cost is a small fraction of device latency and the polling thread idles enough to negate the benefit. Above 64, completion-side pressure grows and the shared ring becomes the contention point – one poll thread per ring, so multiple jobs each need their own ring to parallelise. Cap --numjobs at the poll threads you are willing to dedicate, or you have created more spinners than cores.

Kernel version matters at the guest level: pinning via sq_thread_cpu has been correct since 5.6, while older kernels and Backports-era builds differ in whether the thread inherits SCHED_IDLE and whether it counts against the cgroup CPU quota. Under a vCPU quota, a spinning thread consumes the same budget as application work – so on a plan with a 2 vCPU cap, SQPOLL literally takes a core away from the workload. The instance shape is the deciding constraint; a platform where you get dedicated vCPUs rather than a quota, such as the KVM cloud VPS instances described here, is the only place this trade makes sense.

Deciding With Data

Use SQPOLL when: at least 4 vCPUs, a device that can sustain more than about 50k IOPS at depth, latency target in the tens of microseconds, and a workload that does not go idle for long stretches. Skip it when: 1-2 vCPUs, an overlay or network filesystem where the device is not the bottleneck, or bursty traffic with long compute phases. In those cases plain io_uring with a deep ring already removes most of the per-request syscall cost.

The fallback when SQPOLL is unavailable – no CAP_SYS_NICE, io_uring disabled, older kernel – is to batch submissions with io_uring_submit_and_wait at a depth that keeps a reasonable number of requests in flight per syscall, and to align completions with the application’s own loop rather than polling. On budget instances the instance choice determines the ceiling long before the ioengine does: measure the device’s raw IOPS at depth with fio first, and only then decide whether a dedicated polling thread is buying throughput or just stealing it.

What SQPOLL Actually Removes

The Enter-Idle Path Is What Actually Costs You

Measuring Both Modes With fio

Kernel and Queue-Depth Caveats

Deciding With Data

Related Articles：

Leave a Reply Cancel reply