Enforcing Disk I/O QoS with cgroup v2 io.latency and io.max on Noisy Instances

CPU contention has a scheduler with fair shares and quotas. Disk contention historically had almost nothing: a single backup job saturating the block device could push your database’s read latency from 1 ms to 300 ms while every process on the box appeared to be under a cgroup limit. On cgroup v2, two controllers close that gap. io.latency guarantees a target latency for a workload; io.max imposes hard bandwidth and IOPS ceilings. Together they let you isolate disk access the same way you isolate CPU and memory.

Check That cgroup v2 and the IO Controller Are Live

# Must print cgroup2fs
stat -fc %T /sys/fs/cgroup/

# Confirm the io controller is delegated
cat /sys/fs/cgroup/cgroup.controllers
# expect: cpuset cpu io memory pids ...

# Enable controllers on the root for child use
 echo "+io" > /sys/fs/cgroup/cgroup.subtree_control

If /sys/fs/cgroup reports tmpfs, you are still on cgroup v1 and must boot with systemd.unified_cgroup_hierarchy=1. Most current distributions default to v2, but minimal VPS images occasionally ship hybrid mode. Verify the controller appears in cgroup.controllers before setting any limit – writes to io.max on a cgroup whose parent has not delegated the controller fail silently or with an EACCES, and the limit never applies.

Identify the Device and Its Major:Minor

Every IO limit is per-device, addressed by the device’s major and minor number, not by name. On a VPS the root volume is usually a virtio block device (vda), but additional volumes may be vdb, an NVMe device, or an LVM mapping carrying the dm- major. Reading the wrong number is the most common reason a limit appears to have no effect.

lsblk -o NAME,MAJ:MIN,SIZE,TYPE,MOUNTPOINT
# vda   252:0  80G disk
# └─vda1 252:1  80G part /

# Cross-check against the kernel partition table
 cat /proc/partitions

# Which device backs a given mount
 df -h /var/lib/mysql
 lsblk -no MAJ:MIN $(findmnt -no SOURCE /var/lib/mysql | xargs basename 2>/dev/null) 2>/dev/null || true

Note that limits can be set on either the whole disk (252:0) or a partition (252:1). Setting a limit on a partition throttles only I/O issued to that partition; writes to the raw disk bypass it. For a database with its own dedicated volume, applying the limit to the full device is the safer choice because nothing can accidentally escape the cap.

io.max: Hard Ceilings per Device

io.max accepts per-device limits on read/write bytes per second and IOPS. The four keys are rbps, wbps, riops, and wiops; you may set any subset, and max clears a limit.

# Create a slice for the backup job
systemd-run --slice=backup.slice --unit=nightly-backup --scope true

# Cap the slice: 20 MiB/s write, 2000 write IOPS on device 252:0
 echo '252:0 wbps=20971520 wiops=2000' > /sys/fs/cgroup/backup.slice/io.max

# Read back the effective limits
cat /sys/fs/cgroup/backup.slice/io.max
# 252:0 rbps=max wbps=20971520 riops=max wiops=2000

# Remove one limit
 echo '252:0 wbps=max' > /sys/fs/cgroup/backup.slice/io.max

Setting io.max on a slice applies to every process inside it and to any child cgroups. Because the limit is inherited by children, a runaway worker spawned by the backup job cannot escape the ceiling. This is the correct tool for a bulk job whose only sin is consuming the disk – it needs bytes moved, not the fastest path. Divide the device’s measured throughput, not its theoretical peak, when choosing a number: on a volume that delivers 300 MB/s sequentially, capping a backup at 20 MB/s leaves generous headroom for interactive workloads.

io.latency: Guaranteeing the Critical Path

io.latency works the other way round. Instead of capping the offender, you declare an acceptable latency for the workload that must not degrade, and the kernel throttles anything sharing that device until the target is met. It is a target-based controller: it observes the completion latency of each cgroup’s I/O and applies back-pressure to the cgroups whose target is not the strictest.

# Database slice: keep read latency under 10 ms
 echo '252:0 target=10000' > /sys/fs/cgroup/db.slice/io.latency

# The backup slice tolerates more
 echo '252:0 target=100000' > /sys/fs/cgroup/backup.slice/io.latency

# Inspect the configured targets
cat /sys/fs/cgroup/db.slice/io.latency
# 252:0 target=10000

Values are in microseconds. The lowest target wins priority: the kernel measures observed latency for each cgroup and applies pressure to the others when the target is exceeded. Set a strict target for the database and a generous one for batch work, and the database keeps its 10 ms window even while a backup runs. Note that the strictest target on the device effectively acts as a global latency budget – if you set 10 ms for the database, the controller will throttle the backup regardless of the backup’s own target, which is exactly the isolation you want on a single-volume VPS.

Wiring It Into systemd Services

• IOAccounting=yes on the service and its slice, so systemd-cgtop shows per-service I/O.
• IOWeight= for proportional sharing when you do not need a hard number.
• IOMax= for device-scoped caps in the unit file, expressed as wbps=, riops=, and so on.
• IOSchedulingClass=idle for jobs that should only use leftover disk time.

# /etc/systemd/system/nightly-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
IOSchedulingClass=idle
IOWeight=20
IOMax=252:0wbps=20971520
IOAccounting=yes

# Verify what systemd actually applied
systemctl show nightly-backup.service -p IOWeight -p IOAccounting
systemd-cgtop --order=io

For stateless batch jobs, IOSchedulingClass=idle alone often suffices: the kernel gives the process only leftover disk time and interactive traffic keeps its responsiveness. Reserve io.max and io.latency for workloads that must be fast enough to meet a deadline rather than merely not interfere, and prefer IOWeight when you want proportional sharing instead of a cliff at a hard limit.

Verify the Limits Are Working

# Per-cgroup read/write bytes, IOPS, and discarded I/O
cat /sys/fs/cgroup/backup.slice/io.stat
# 252:0 rbytes=... wbytes=... rios=... wios=...

# Stall information for the cgroup
cat /sys/fs/cgroup/backup.slice/io.pressure

# Live per-device throughput and queue depth
iostat -x 1 | grep -E 'Device|vda'

If io.stat shows the backup slice writing far below its cap, check that the limit landed on the correct major:minor and that the process is genuinely inside that slice – a job launched by cron inherits the cron slice, not the one you built for it. Move it under systemd or use systemd-run --slice= from the cron job itself. A common trap is launching the daemon from an interactive shell: the process inherits the login session’s cgroup and your carefully constructed slice sits empty.

Limits, Not Miracles

Disk QoS is a contention-management tool, not a capacity upgrade. On an instance whose storage tier is already saturated by host-level noisy neighbours, guest-side limits cannot manufacture IOPS that are not offered. Measure raw device latency with fio first, and if the ceiling is set by the provider, move the workload to a faster volume class. Our managed versus unmanaged service comparison covers where that kind of host-side isolation is included, and the VPS provisioning guides discuss selecting storage by measured IOPS rather than headline type. Applied together, io.latency and io.max turn disk contention from an unpredictable, machine-wide problem into a configured policy: the critical path keeps its latency target, the batch jobs finish on budget, and neither one has to know the other exists.