nftables Forwarding Rate: Conntrack Cost, Sets and Flowtables

Five routes cost more than the whole nftables ruleset: the conntrack entry, mangle prerouting, mangle postrouting, forward filter and nat postrouting. A router with a 500-line zone-based firewall still spends most of its per-packet budget on that fixed tax, because conntrack evaluates every new flow and every established flow is still hashed and looked up. This article measures which parts actually cost what on a 1 vCPU / 1 GB KVM guest, and which knobs have a real effect versus which are folklore.

Concrete baseline: on a 1 vCPU vm.small KVM instance with virtio-net and a 1500-byte MTU, a plain forwarding path without firewalling moves about 900 Mbit/s at ~140k packets per second. Adding a stateful nftables ruleset with four base chains drops that to roughly 550–650 Mbit/s at ~90k pps. The next sections measure the gap and attack it in order.

Build a repeatable packet-rate harness first

Throughput in Mbit/s hides packet-rate problems. Use a fixed packet size and measure packets per second at both ends, so you can tell whether a change moved bandwidth or moved CPU per packet.

# generator side (a second VPS or any LAN host)
pktgen or: iperf3 -c 10.0.0.2 -u -b 0 -l 64 -t 30 --json

# device under test: per-CPU packet counters and softirq cost
mpstat -P ALL 1 30 | awk '/Average/{print}'
cat /proc/net/softnet_stat | awk '{print $1, $2, $3}' | head -4
nstat -az | grep -E 'IpInReceives|IpOutRequests|IpInHdrErrors'

/proc/net/softnet_stat column 2 is dropped and column 3 is time_squeeze — the number of times ksoftirqd ran out of its budget without draining the backlog. Rising time_squeeze at line rate means the per-packet cost is already at the limit of one core, and further rule optimisation will only help if it is measured against that ceiling. Record the number before and after every change in this article.

Isolate conntrack from the ruleset

Start with three configurations and record pps for each: no netfilter at all, filter-only rules with ct state matches, and a full NAT setup. The difference between the first two is conntrack plus rule traversal; the difference between the second and third is NAT.

# filter-only, no NAT
nft -f - <<'NFT'
table inet fw {
  chain fwd { type filter hook forward priority filter; policy accept
    ct state established,related accept
    tcp dport { 22, 443 } accept
    drop
  }
}
NFT

# then add NAT and re-measure
nft add table ip nat
nft 'add chain ip nat post { type nat hook postrouting priority srcnat; }'
nft add rule ip nat post oifname "eth0" masquerade

Expected shape on a 1 vCPU guest: filter-only costs 10–20 % of forwarding pps, NAT costs another 15–25 % on top, and the cost scales with the number of rules only until the first match. That last point is the one people get wrong — a 5,000-line set of literal ip daddr rules is not 1,000× slower than five rules if you convert it to a set. It is slower because each rule is a separate memory access.

Convert literal rules to sets and maps

A rule with a literal address is evaluated by a linear comparison. A rule with a set is evaluated by a hash or rbtree lookup whose cost barely changes between 10 and 100,000 entries. For anything above roughly 20 elements the set form wins outright.

table inet fw {
  set allow_v4 { type ipv4_addr; flags interval
                 elements = { 203.0.113.0/24, 198.51.100.7, 192.0.2.0/28 } }
  set bulk_ports { type inet_service; flags interval
                   elements = { 10000-10100, 20000-20100 } }

  chain fwd {
    type filter hook forward priority filter; policy drop
    ct state established,related accept
    ip saddr @allow_v4 tcp dport @bulk_ports accept
    iifname "wan0" oifname "lan0" accept
  }
}

The flags interval declaration is what enables prefix and range storage as a tree rather than one hash entry per address. Without it, 192.0.2.0/24 expands to 256 entries and the set becomes large and slow to update. Check the working set size with nft list set inet fw allow_v4 and its memory with nft -j list ruleset | jq if you need numbers.

Where flowtables help and where they do not

A flowtable moves established-flow forwarding out of the normal netfilter path entirely: after the first packet of a flow is classified, subsequent packets are fast-forwarded at a point that skips routing and most hooks. Used by the bigger routers for a decade, it is now straightforward in nftables and worth enabling on any VPS acting as a router or VPN gateway.

nft add table inet fast
nft 'add flowtable inet fast f { hook ingress priority 0; devices = { eth0, eth1 }; }'
nft 'add chain inet fw fwd { type filter hook forward priority filter; }'
nft add rule inet fw fwd ip protocol { tcp, udp } flow add @f

# verify flows are being offloaded
nft list flowtable inet fast
conntrack -L -o extended | head

Two constraints matter. First, the flowtable only covers traffic passing through the device — locally generated or locally terminated traffic never sees it, so it does nothing for a plain web server. Second, hardware offload is a guest-level non-starter on a VPS: virtio-net does not expose the switching hardware, so you get the software fast path only. That still removes most of the per-packet cost, typically 20–40 % more forwarding pps on a 1 vCPU guest, without the multi-gigabit gains seen on bare metal.

The sysctls that matter for forward rate

Three parameters govern how a forwarding guest handles bursts. All three are cheap to change and easy to measure against the time_squeeze counter from the harness earlier.

ParameterDefaultForwarding valueEffect
net.core.netdev_max_backlog10005000–16000fewer drops on RX bursts
net.core.somaxconn40964096 (no change)irrelevant for forwarding
net.ipv4.ip_forward01required
net.netfilter.nf_conntrack_max65536 or RAM-derived2–4× expected flowsavoids table-full drops
net.core.dev_weight64128–256larger RX poll budget per softirq

Raising dev_weight increases how many packets one NAPI poll processes before yielding, which matters when a single receive queue serves all traffic. On a virtio-net device with multiple queues, check ethtool -l eth0 first — spreading interrupts across vCPUs usually beats a bigger poll budget, and on a 1 vCPU plan you have nothing to spread to.

Ruleset shape: what to put first

Rule order in nftables is evaluated top to bottom within a chain, so the first rule should be the one that matches the overwhelming majority of packets. On a router that is ct state established,related accept. Everything else — new connections, ICMP, logging — can be arbitrarily expensive because it sees a tiny fraction of traffic. A common misconfiguration puts per-protocol anti-spoofing rules above the established rule, which forces every packet through a dozen comparisons.

Measure it with the counters nftables keeps per rule; they are free until you read them:

nft add rule inet fw fwd ct state established,related counter accept
nft list chain inet fw fwd   # shows "packets N bytes M" per rule

# order check: after 60s, the first counter should hold >95% of all packets
nft list chain inet fw fwd | grep counter

If the established rule is not absorbing the vast majority, your connection churn is unusually high — check conntrack -S for insert failures and nstat for TcpAttemptFails. High churn turns a forwarding problem into a conntrack table sizing problem, which is a different fix entirely.

A build order that leaves a measurable trail

  • Capture baseline pps at 64 and 1500 bytes with no ruleset; record time_squeeze.
  • Add the filter chain with the established rule first; measure again.
  • Add NAT; measure. If the drop is under 10 %, stop — NAT is not your bottleneck.
  • If you forward for a LAN or VPN, add the flowtable; expect 20–40 % more pps on a 1 vCPU guest.
  • Convert any rule with more than 20 literal addresses or ports to a set with flags interval.
  • Set netdev_max_backlog and dev_weight; re-measure under a 10× burst to see the drop count move.
  • Check queue counts with ethtool -l and /proc/interrupts; only then consider more vCPUs.

Keep each intermediate ruleset in a file and reload with nft -f rather than editing live. If a change makes things worse, you want to go back one step, not reconstruct the previous state from shell history. The conntrack table survives a ruleset reload, so flush it explicitly (conntrack -F) between throughput tests or you will be measuring a warm table against a cold one.

When to stop optimising the software path

Once time_squeeze is flat at line rate and the flowtable is enabled, the remaining ceiling is the virtio datapath and the single core running ksoftirqd. At that point the options are multi-queue virtio (more vCPUs, queue-per-CPU), a plan with a larger per-core allocation, or moving routing to a dedicated appliance. If the workload is latency-sensitive rather than throughput-sensitive — a VPN hub, a game relay, a small office gateway — the packet rate rarely reaches the ceiling anyway and the effort belongs on buffer and qdisc tuning instead, which is the subject of choosing between shared and dedicated resources for different traffic shapes.

For context on why the guest sees this ceiling at all, the cloud VPS scaling model explains how provider-level port shaping and per-interface caps interact with what you measure inside the guest: on a small plan the shaping is frequently the visible limit long before the CPU is.

Leave a Reply