A single-vCPU VPS can be perfectly healthy by every CPU metric and still drop packets under load. The usual cause is that the receive path is processed by softirqs on whichever CPU the driver raised them on, and on a one-vCPU guest there is exactly one place for that work to happen. This is not the same problem as a full softirq backlog; it is about where the work executes, how many queues exist, and whether the one CPU can keep up. This article covers the measurements that prove the bottleneck is in the softirq path, and the knobs that move it.
Hardirq vs Softirq: The Split That Matters
When a frame arrives, the device raises an interrupt. The hardirq handler runs with interrupts disabled and is supposed to do the minimum: acknowledge the device, mask further interrupts, and schedule the deferred part. The deferred part is a softirq — NAPI polling in the NET_RX vector — which runs with interrupts enabled, either right after the hardirq returns or in the ksoftirqd thread when load is high.
The same received packet can therefore be charged to %irq, %softirq, or to a kernel thread. The measurement question is how much CPU the receive path consumes per unit of traffic, and whether that work competes with your application threads.
mpstat -P ALL 2 5
grep -E '^(cpu|intr)' /proc/stat
ps -eo pid,comm,pcpu | grep -E 'ksoftirqd|kworker'
cat /proc/interrupts | head -20On a one-vCPU guest, %softirq climbing while %sys stays low is the signature to look for. Because there is only one CPU, ksoftirqd/0 competing directly with your application threads is expected under sustained traffic, and it shows up as reduced application throughput rather than as a network error.
Confirm the Bottleneck Exists
Before tuning, prove that the ceiling is on the receive path and not on the sender, the link, or your application. The clean test is to generate traffic from a separate host and compare the rate at which packets arrive with the rate at which your process consumes them. If RX packets rise while your application’s request rate plateaus, and %softirq saturates on the single vCPU, the receive path is the limit.
# on the VPS: receive-side counters, sampled
awk 'NR>2 {print $1, "rx_pkts="$3, "rx_drop="$4, "rx_err="$5}' /proc/net/dev
ss -lntu | head
nstat -az | grep -E 'InErrors|InDrops|RcvbufErrors|ListenDrops|TCPRcvQDrop'Where the Interrupts Land
A virtio NIC in a KVM guest can present one or more virtqueues. Each queue can be bound to a vCPU, and in multi-queue mode each virtqueue’s interrupt has its own affinity. The mapping is visible in two places: the queue directories under the device, and the interrupt affinity mask in /proc/interrupts. Reading both is the fastest way to see whether all RX work is funnelled onto one CPU.
ls /sys/class/net/eth0/queues/
ethtool -l eth0
ethtool -S eth0 | head -30
for irq in $(awk '/virtio/{print $1}' /proc/interrupts | tr -d :); do
echo "$irq: $(cat /proc/irq/$irq/smp_affinity_list)"
doneOn a one-vCPU instance every mask will read 0, and that is the point: there is nowhere else to move the work. Interrupt affinity tuning is therefore not available, and what remains is reducing the cost per packet or adding a CPU. On instances with two or more vCPUs, spreading queue interrupts across vCPUs is the first and most effective fix.
RPS and RFS: Spreading Work Without Hardware Queues
Receive Packet Steering (RPS) emulates multiple receive queues in software by hashing flows across CPUs and queueing the deferred processing on a per-CPU backlog. Receive Flow Steering (RFS) goes further and steers a flow’s processing to the CPU where the application thread that owns the socket is running, improving cache locality.
On a single-vCPU guest neither does anything useful — there is one CPU to steer to — but the configuration still matters when you later resize. On a 2- or 4-vCPU plan, RPS is often the difference between one saturated core and even distribution, particularly when the virtio device exposes only one receive queue.
# RPS: bitmask of allowed CPUs, one value per rx queue
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus # CPUs 0-3
# RFS: global flow table size and per-queue flow limit
sysctl -w net.core.rps_sock_flow_entries=32768
echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
# verify the packets actually got steered
cat /proc/softirqs | grep NET_RX
mpstat -P ALL 2 5When One Queue Is the Ceiling
The definitive sign of a single-queue ceiling is that throughput stops scaling while packets keep arriving. In a guest this appears as: one virtqueue active in ethtool -S, RX drops rising, %softirq pinned near the CPU’s capacity, and application throughput flat. A useful check is to compare packets-per-second at two different frame sizes; if small frames collapse the rate far more than large ones, you are limited by per-packet processing cost rather than by bandwidth.
ethtool -g eth0
ethtool -G eth0 rx 1024 tx 1024
# batching indicators
ethtool -S eth0 | grep -iE 'packets|drops|queue'
cat /proc/softirqs | awk '/NET_RX/{print}'A larger ring absorbs bursts, but it does not reduce steady-state CPU per packet: per-packet cost is dominated by descriptor handling, skb allocation and the stack walk, which is why a flood of small messages hits a hard ceiling that large frames never reach. Larger rings buy burst headroom; more vCPUs buy packets per second.
Putting the Evidence Together
A convincing diagnosis measures five things over one window: the per-CPU NET_RX delta from /proc/softirqs, the %softirq share from mpstat, the drop counters from nstat, the queue count from ethtool -l, and the application’s achieved request rate. When the softirq delta grows linearly with arriving packets while the request rate plateaus, the receive path is the bottleneck and no other explanation fits.
From there, the fix depends on what the platform allows. More vCPUs plus queue interrupts spread across them is the real solution; RPS helps when the driver exposes a single queue; coalescing and larger rings buy headroom. When none of the kernel levers is available — no multi-queue, no coalescing control, one vCPU — the constraint is the plan, and the VPS plans with more vCPUs and dedicated network queues are the practical next step. Some of the apparent network limit is really storage or scheduler contention on the same core, covered in the kernel tuning guide for governors, swappiness and I/O schedulers.
Quick Diagnostic Sequence
mpstat -P ALL— is%softirqthe dominant term on the single vCPU?/proc/softirqs— doesNET_RXgrow on one CPU only?nstatand/proc/net/dev— are drops at the socket, queue, or device level?ethtool -land/proc/interrupts— how many queues, and where are their interrupts?- Vary frame size — does small-packet rate collapse relative to large?
- Enable RPS/RFS only after confirming the bottleneck is receive-side CPU.
On a single-vCPU VPS the softirq path is the only place receive work can run, so it saturates first and saturates quietly. Measuring where packets are counted, where interrupts land, and how the drop counters move is enough to prove the bottleneck without guesswork — and to know whether it is fixable from the guest or only by changing the plan.
Leave a Reply
You must be logged in to post a comment.