TIME_WAIT Port Exhaustion on a Busy VPS: Measuring It and Tuning Safely Without Breaking NAT

A VPS with 65,535 ephemeral ports and high request churn runs out of four-tuples before it runs out of bandwidth, CPU or file descriptors. Every outbound connection from your nginx upstream pool, health check or PHP-FPM-to-Redis call consumes a local port in ESTABLISHED, then parks it in TIME_WAIT for 60 seconds after close. At 1,500 new connections per second to a single backend IP:port, the window closes long before load average moves.

The symptom looks like an application bug: requests fail with Cannot assign requested address, or a client throws connection refused to a backend that is demonstrably listening, spiky rather than sustained. Because the ephemeral range is consumed per destination address, a service talking to one database IP exhausts far faster than the raw port count suggests. If you are still sizing the instance for this workload, the spec-sheet reading guide explains which host limits actually bind.

Why TIME_WAIT Exists and Why 60 Seconds Matters

TIME_WAIT is not a leak. After the side that sent the first FIN completes the close handshake, it holds the four-tuple for twice the maximum segment lifetime — 60 seconds on Linux, fixed as TCP_TIMEWAIT_LEN and not tunable. The window absorbs retransmitted duplicate FINs so they cannot corrupt a new connection, and guarantees a delayed segment from the old connection cannot reach the new one.

The consequence is a hard ceiling per destination: at most range / 60 new connections per second to any single IP:port once the pool saturates. With the default ip_local_port_range of 32768-60999 that is 28,232 ports, or about 470 connections per second to one destination. Real services fail well below that, because the range is shared across every outbound connection on the host.

Measuring It: Aggregate First, Then Per Destination

Start with ss -s, which prints a one-screen summary including a timewait count beside the socket totals. Thousands on a quiet box is normal; tens of thousands with a climbing ephemeral-port count is the signature. Then break it down by destination, because one saturated peer is a different problem from a globally exhausted range.

The second measurement is the range itself: how many ports in ip_local_port_range are currently bound to destination that matters. Above roughly 60% utilisation, new connects fail intermittently as the kernel cannot find a free port not in TIME_WAIT for that target address.

Counting TIME_WAIT Properly

The commands below produce aggregate state, a per-peer ranking, the ephemeral range, and the sysctls that matter. Run them together: a large TIME_WAIT count with low connection churn is normal keep-alive behaviour; the same count with high churn is exhaustion.

# 1. Aggregate socket state, including timewait
ss -s

# 2. TIME_WAIT broken down by destination (top peers)
ss -tan state time-wait | awk 'NR>1 {split($5,a,":"); print a[1]":"a[2]}' | sort | uniq -c | sort -rn | head -15

# 3. Count TIME_WAIT against total sockets
echo "timewait=$(ss -tan state time-wait | wc -l) total=$(ss -tan | wc -l)"

# 4. Ephemeral range and how much of it is in use
sysctl net.ipv4.ip_local_port_range    # default 32768-60999 = 28232 ports

# 5. Relevant sysctls and conntrack headroom
sysctl net.ipv4.tcp_tw_reuse net.ipv4.tcp_max_tw_buckets net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
dmesg | grep -i 'nf_conntrack: table full' | tail -5

# 6. Do not do this: removed from the kernel and harmful behind NAT
# sysctl -w net.ipv4.tcp_tw_recycle=1

Read step 4 carefully. If the TIME_WAIT count from step 3 is a meaningful fraction of the 28,232-port default, you are capacity-bound and retrying in the application will not help. Step 5 is the one people skip: if the host’s conntrack table is full, packets drop before TIME_WAIT is relevant, and guest sysctls will not fix it.

The Tunables, Ranked by Safety

The safest effective change is widening the range: net.ipv4.ip_local_port_range = 10240 65535 yields about 55,000 ports, nearly doubling the per-destination ceiling. It costs nothing, is immediately reversible and does not alter wire behaviour. Second safest is net.ipv4.tcp_tw_reuse = 1, which lets outbound connections reuse a TIME_WAIT socket when the kernel judges the timestamp strictly newer.

Note the asymmetry: tcp_tw_reuse applies to outbound connections only. It does nothing for a server accepting connections on a listening port, and nothing when the peer dictates closure. If your VPS is the server, the fix is keep-alive and pooling, not this sysctl.

What Breaks Your NAT: tw_recycle and Buckets

net.ipv4.tcp_tw_recycle was the classic fix and is now a hazard. It used per-host PAWS timestamp checks to skip TIME_WAIT, dropping connections from clients behind NAT that present non-monotonic timestamps — one common NAT device and every client behind it fails intermittently. It is removed in recent kernel trees and ignored when set. Remove any line enabling it.

net.ipv4.tcp_max_tw_buckets caps how many TIME_WAIT sockets the system keeps; beyond the cap the kernel destroys the oldest and logs time wait bucket table overflow. Lowering it to “reclaim” ports is wrong: it discards TIME_WAIT protection, admits stale segments into new connections, and the counter is global rather than per-destination. Leave it at default or raise it to stop log spam.

The Real Fix Is Fewer Connections

Every sysctl above buys headroom in a fixed budget. The structural fix is to stop creating a connection per request: enable upstream keep-alive with upstream keepalive 32;, proxy_http_version 1.1 and proxy_set_header Connection "", pool database connections, and set client timeouts so idle sockets are reused. Cutting churn by 90% removes the problem instead of deferring it.

Action	Risk	Effect
Widen ip_local_port_range	None	~2x per-destination ceiling
tcp_tw_reuse=1	Low, outbound only	Reuses TIME_WAIT for new outbound
Upstream keep-alive/pooling	None	Removes 80-95% of churn
Lower tcp_max_tw_buckets	High	Drops TIME_WAIT protection
tcp_tw_recycle=1	Breaks NAT clients	Removed from kernel; ignored

Measure after each change. A healthy post-fix picture is TIME_WAIT that plateaus instead of climbing, a per-destination table with no dominant peer, and no Cannot assign requested address across a full traffic cycle. For the syscall-layer view of a busy box under connection load, io_uring sqpoll versus interrupt-driven I/O applies the same measure-before-tuning method.

Detecting It Before Users Do

Expose two numbers to monitoring: TIME_WAIT sockets per destination, and the ratio of used ephemeral ports to range size. Alert at 70% of range for any single destination and at 20,000 total TIME_WAIT sockets. Both are cheap to collect from ss every minute and move slowly enough that you see a trend rather than an outage. Persist tuned values in /etc/sysctl.d/ so a reboot cannot silently revert them.

Also watch conntrack on the virtualisation host where your provider exposes it; a guest can be perfectly tuned and still lose packets when the parent’s nf_conntrack_max fills. When the numbers disagree, trust the one corresponding to where the failure actually happens.