Reading /proc/net/softnet_stat Correctly: Diagnosing Softirq Pile-Up on Virtio NICs

The ninth column of /proc/net/softnet_stat is the most misread number in Linux network troubleshooting. It counts the times a CPU’s softirq handler exhausted its NAPI poll budget with packets still queued — time_squeeze. A steadily rising value on one CPU while the others sit idle is the definitive signature of a single-threaded network receive path, and it is the reason a 4 vCPU VM can drop packets at 40% total CPU utilisation.

Almost every “the network is slow but the CPU looks fine” report fails at step one: the counters in that file are hexadecimal, printed without a 0x prefix. The value 10 in column nine is sixteen squeezes, not ten. The value 1a is twenty-six. Reading them as decimal is a systematic error that underestimates pressure by up to ten times on the digits that matter.

Parse the file correctly before believing anything

Each line is one CPU, in order. Column 1 is processed packets, column 2 is dropped packets, column 9 is time_squeeze, and column 11 (on newer kernels) is cpu_collision. Every field is a bare hex string, and the line order corresponds to CPU numbering with no explicit label, which matters on systems with sparse CPU numbering inside a cpuset.

cat /proc/net/softnet_stat
# columns are hex: 1=processed 2=dropped 9=time_squeeze 11=cpu_collision

python3 - <<'PY'
import time
def snap():
    out=[]
    for i,line in enumerate(open('/proc/net/softnet_stat')):
        f=line.split()
        out.append((i,int(f[0],16),int(f[1],16),int(f[8],16)))
    return out
a=snap(); time.sleep(10); b=snap()
print(f"{'cpu':>3} {'pkt/s':>10} {'drop/s':>8} {'squeeze/s':>10}")
for x,y in zip(a,b):
    print(f"{x[0]:>3} {y[1]-x[1]:>10} {y[2]-x[2]:>8} {y[3]-x[3]:>10}")
PY

The deltas over a 10-second window are what you interpret, never the cumulative totals. A box that has been up for 200 days will always show frightening absolute numbers; what matters is whether the counters are still moving.

What the columns actually mean

Column	Field	Meaning	Healthy
1	processed	Packets passed up the stack	rising with traffic
2	dropped	Dropped at the backlog before the stack saw them	0
9	time_squeeze	Out of NAPI budget with work remaining	0 under normal load
11	cpu_collision	Two CPUs contended for the same queue	0

time_squeeze is not a drop. It means the kernel stopped processing a queue because it used its full budget for that scheduling slice, and will resume later. Single squeezes are harmless. Thousands per second on one CPU mean the queue is deep enough that latency accumulates inside the kernel before the packet ever reaches your application — TCP clients see it as jitter and retransmits, not as throughput loss, which is why the symptom is so often misdiagnosed as an application bug.

The single-queue virtio bottleneck

Default virtio-net devices in KVM present a small number of receive queues — often one. With one queue, all incoming packets for the guest are delivered to a single vCPU’s softirq context, regardless of how many vCPU the guest has. That CPU saturates its network path long before the guest reports high overall CPU usage, because top shows the softirq work as system time concentrated on one core, and averages it away in the summary line.

# how many queues did the guest get, and on which CPUs?
ethtool -l eth0
ethtool -S eth0 | egrep 'queue_[0-9]+_(rx|tx)_packets' | head -20
# per-CPU softirq breakdown, watch the NET_RX column
watch -n1 'grep NET_RX /proc/softirqs'
# or use mpstat: %soft on one core climbing while others idle
mpstat -P ALL 2 5

If ethtool -l reports Combined: 1 and only queue_0 counters are moving, the guest has a single receive path. mpstat will show one CPU with %soft above 30 while the rest are idle — the exact pattern that time_squeeze counts.

The fix has two halves. Inside the guest you can request more queues, but the device must have been created with them. Check whether the host exposed multiple queues first:

# request 4 combined queues (fails if the device has fewer)
ethtool -L eth0 combined 4
ethtool -l eth0
# spread RX processing across cores
systemctl enable --now irqbalance
grep eth0 /proc/interrupts | head

On a provider that passed mq=on,vectors=8 to QEMU, this works and time_squeeze stops accruing on a single core. On a provider that did not, ethtool -L returns “Cannot set device channel parameters: Invalid argument.” That error is not a guest misconfiguration — it is an architectural limit of the plan, and the only remedy is a plan change or an SR-IOV or virtio-net multi-queue instance. Checking this before renting is worth more than any amount of in-guest tuning; the VPS network and hardware notes on the main site describe which tiers expose multi-queue virtio.

Backlog queues: when rps and rps_flow_cnt help

If multi-queue is unavailable, the kernel can still distribute processing with Receive Packet Steering. RPS hashes flows across CPUs in software, which costs extra cycles per packet but converts a single-core bottleneck into a multi-core one. It is a net win above roughly 200k pps on a 4 vCPU guest and a small loss below it.

# enable RPS on a 4-vCPU guest: CPU mask f = cpus 0-3
for q in /sys/class/net/eth0/queues/rx-*/rps_cpus; do echo f > $q; done
# flow hash table, apply to all sockets if unsure which protocol saturates
sysctl -w net.core.rps_sock_flow_entries=32768
for q in /sys/class/net/eth0/queues/rx-*/rps_flow_cnt; do echo 32768 > $q; done

Bind the softirq kernel threads to specific cores so they do not migrate into the application’s cores. On a 2 vCPU guest there is nothing to distribute and RPS is not worth enabling; the intervention is a larger guest.

# pin ksoftirqd to a dedicated core on a 4+ vCPU guest
systemctl stop irqbalance
for pid in $(pgrep ksoftirqd); do taskset -pc 3 $pid; done
# raise the socket backlog: must exceed netdev_max_backlog or you trade drops
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.core.somaxconn=4096

Note the relationship between netdev_max_backlog and the drop counter in column 2. When the backlog is full, packets are dropped before softirq ever runs, and column 2 rises while column 9 stays flat. That combination tells you to raise the backlog rather than add queues — a distinction that saves a lot of wasted effort.

Which signature means what

Column 9 climbing, one CPU at 100% %soft, others idle — single-queue receive path. Request more queues; if unavailable, add RPS or move to a multi-queue plan.
Column 2 climbing, column 9 flat — userspace or socket backlog overflow. Raise net.core.netdev_max_backlog and net.core.somaxconn, then check application accept loops.
Both climbing together on all CPUs — the guest is genuinely out of CPU for the packet rate. Reduce packet rate with GRO/GSO hints or move traffic behind a CDN; no kernel knob fixes this.
Column 11 non-zero — multiple CPUs are colliding on one queue, which only happens with misconfigured RPS masks or a badly pinned application.

Confirm the diagnosis with a real load

Counters under production load are the only meaningful test. If you cannot reproduce the traffic, generate it from another host with iperf3 -u -b 0 or a SYN flood replacement like hping3 --flood -S -p 80 against a scratch instance. Watch time_squeeze and %soft on the receiving guest simultaneously.

# receiving side, one terminal
watch -n1 'awk "{print \$9}" /proc/net/softnet_stat | while read h; do printf \"%d \" 0x\$h; done; echo'
# and in another
mpstat -P ALL 1

At saturation you will see the squeeze counter accelerate sharply at a specific packet rate — typically 150k–400k pps for a single virtio queue depending on packet size. That knee is the true capacity of the instance’s network path, and it is usually well below the advertised port speed in packets per second even when the bit rate looks fine.

Frame size changes the pps budget

Softirq cost is per packet, not per byte. A 64-byte flood costs the same CPU as a 1,400-byte one, so a guest that comfortably handles 900 Mbps of bulk transfer can collapse at 900 Mbps of small requests. This is why API servers with chatty protocols saturate before file servers do. Enabling GRO on receive and GSO on transmit amortises the per-packet cost when the traffic allows it:

ethtool -K eth0 gro on gso on tso on
ethtool -k eth0 | egrep 'generic-receive|tcp-segmentation|generic-segmentation'

These are on by default on most distributions, but some provider images disable them. Verifying costs ten seconds and occasionally recovers 20–30% of achievable throughput.

The general lesson: when a guest’s network misbehaves, the first file to read is /proc/net/softnet_stat, the first mistake to avoid is reading hex as decimal, and the first architectural question to ask is how many receive queues the plan was provisioned with. Everything else — RPS, backlog sizes, core pinning — is compensation for a queue count you cannot change from inside the guest. If you are still selecting an instance, the provider comparison on the main site lists virtualization and network device details per plan, and our network verification guide covers the throughput tests that pair with these counters.