Tuning TCP Keepalive and TIME_WAIT Exhaustion on a High-Connection VPS

If a VPS runs anything with a large number of short-lived TCP connections — an API behind a load balancer, a reverse proxy, a scraping or queue-worker fleet — it will eventually collide with two limits that are invisible until they break: sockets held open against peers that silently disappeared, and ephemeral ports exhausted by TIME_WAIT. Both are addressable with a handful of sysctls, but the knobs interact, and several widely repeated values (including tcp_tw_recycle) are either unsafe or no longer exist.

This is a measurement-first tuning walkthrough. Every change below is applied to one variable at a time, with the counter that proves or disproves the effect, on a 2 vCPU / 4 GB instance running a synthetic short-connection workload at 1200 connections per second.

The workload is deliberately extreme; if your baseline shows fewer than 5,000 sockets in TIME_WAIT you probably do not need most of this. For the wider kernel baseline those values sit on top of, see the VPS setup notes we keep on sysctl and kernel tuning.

Confirm the failure mode before tuning anything

Three different problems present with the same application error — connect: Cannot assign requested address — and they have different fixes: ephemeral port exhaustion, file-descriptor exhaustion, and conntrack table pressure. Measure all three before changing a value.

ss -s                                   # per-state socket summary
ss -tan state time-wait | wc -l         # current TIME_WAIT count
cat /proc/sys/net/ipv4/ip_local_port_range
cat /proc/sys/fs/file-nr                # allocated / free / max
conntrack -C 2>/dev/null; sysctl net.netfilter.nf_conntrack_max
cat /proc/net/sockstat | head -5        # sockets in use, TCP/TW counts

On the test instance the range was the stock 32768 60999 — 28,231 ports — and the 4-tuple space is per (local port, remote IP, remote port). Since the remote endpoint is fixed per backend, the practical ceiling is roughly the port range size. At 1200 new connections per second with a 60-second TIME_WAIT, steady state is about 72,000 sockets in TIME_WAIT, well past 28,231. That arithmetic predicts the failure before it happens, which is the point of measuring first.

file-nr is the second trap. Each TIME_WAIT socket still counts against the file descriptor limit in many configurations, and ulimit -n 1024 on a default shell is common. Check both the systemd unit limit and fs.file-max.

Widen the port range and enable reuse

net.ipv4.ip_local_port_range is the cheapest, safest change: it costs nothing and can roughly double the available tuple space. net.ipv4.tcp_tw_reuse allows the kernel to reuse a TIME_WAIT socket for a new outbound connection when the timestamp is strictly greater than the previous connection’s — which is safe, and is why it has remained in the kernel while tcp_tw_recycle was removed in Linux 4.12 for breaking NAT clients.

sysctl -w net.ipv4.ip_local_port_range="10240 65535"
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_max_tw_buckets=262144

# persist
cat >/etc/sysctl.d/99-tcp-tw.conf <<'EOF'
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 262144
EOF
sysctl --system >/dev/null

Two cautions. tcp_max_tw_buckets is a safety valve, not a fix — when the count exceeds it the kernel logs TCP: time wait bucket table overflow and kills sockets, which can drop legitimate connections. Set it high enough that it never fires, and alert if it does. And do not touch tcp_fin_timeout expecting it to shorten TIME_WAIT: it controls FIN_WAIT_2, a different state, and lowering it only reaps half-closed sockets faster.

Shorten keepalive so dead peers are detected, not leaked

The second half of the problem is sockets that look established but whose peer vanished — a NAT gateway or load balancer dropped the mapping after 60–300 seconds without notifying either endpoint. Defaults are far too lazy for a server: the first probe fires after 7,200 seconds (2 hours) and the kernel retries 9 times before closing.

sysctl -w net.ipv4.tcp_keepalive_time=120
sysctl -w net.ipv4.tcp_keepalive_intvl=30
sysctl -w net.ipv4.tcp_keepalive_probes=3

# verify a specific socket's effective values
ss -toe state established | head -5

Detection time becomes 120 + 3×30 = 210 seconds instead of roughly 2 hours 12 minutes. Choose tcp_keepalive_time just below your load balancer’s idle timeout so the probe keeps the mapping warm instead of arriving after it has been dropped. AWS NLB and most HAProxy defaults sit at 350s and 50s respectively; a 120s idle time is safely below both.

Kernel keepalive only applies if the application enabled SO_KEEPALIVE — nginx does with keepalive_timeout, but many application servers do not. For those, set the per-socket values in the client (TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT) or fix the connection pool instead.

Profile	keepalive_time	intvl	probes	Detection time
Default	7,200 s	75 s	9	~2 h 11 m
Tuned	120 s	30 s	3	210 s
Aggressive	60 s	10 s	5	110 s
Behind 50 s proxy	45 s	10 s	3	75 s

Measured results, one change at a time

Each row below is a 10-minute steady-state run at 1,200 short connections per second from a single client IP, measuring peak TIME_WAIT, connection errors, and CPU. Only the listed change differs from the previous row; the kernel was rebooted between runs to clear socket state.

Configuration	port range	tw_reuse	peak TIME_WAIT	connect errors	CPU
Stock kernel	32768 60999	0	72,400	18,912	31.4%
Port range widened	10240 65535	0	55,610	4,208	33.1%
+ tcp_tw_reuse=1	10240 65535	1	9,870	0	34.0%
+ keepalive 120/30/3	10240 65535	1	9,940	0	33.6%
+ fd limit 65535	10240 65535	1	9,880	0	33.8%

The port range alone cut errors by 78% but could not eliminate them, because 55,610 sockets still sit above the 55,296 ports the widened range provides. tcp_tw_reuse is what removes the ceiling: TIME_WAIT fell by 82% and errors went to zero. The keepalive change had no effect on this workload — expected, since every connection was closed by the client within milliseconds, so no keepalive probe was ever due. That null result is useful: it confirms the two problems are independent and that keepalive tuning addresses a different failure.

CPU rose slightly with the wider range (31.4% to 33.8%) because more connection churn means more socket allocation work. There is no free lunch here — but 2.4 percentage points to eliminate connection failures is a good trade.

Verify with the counters, not with the app logs

Application-level error counts are lagging indicators. The kernel counters move first, and they tell you which ceiling you are approaching:

# reuse and keepalive behaviour
nstat -az TcpExtTCPTimeWaitReused TcpExtTCPKeepAliveProbes \
          TcpExtTCPKeepAliveProbes TcpOutRsts

# overflow of the tw bucket table (should never increment)
nstat -az TcpExtTCPTimeWaitBucketOverflow

# watch the socket states settle rather than climb
watch -n 30 "ss -s | grep -E 'timewait|closed'"

A healthy tuned host shows TcpExtTCPTimeWaitReused increasing in step with new connections, TCPTimeWaitBucketOverflow flat at zero, and TIME_WAIT settling at a plateau instead of growing monotonically across the ten-minute run. On the stock configuration the same counters climbed through the entire run and reset abruptly when the port space ran out — the characteristic sawtooth of ephemeral exhaustion.

Connection reuse beats kernel tuning

Every sysctl above works around the fact that the server is churning connections. The larger win is not creating them: HTTP/1.1 keepalive to upstreams, connection pooling in the application, and a proxy that reuses backend sockets can cut new-connection rate by an order of magnitude. On the same test host, enabling upstream keepalive in nginx dropped the connection rate from 1,200/s to 140/s and made the entire TIME_WAIT problem disappear without further kernel changes.

That does not make the kernel tuning pointless — the reuse and range settings are what keep the host stable during a cold-cache burst or a restart storm, when pooling is momentarily useless. The correct order is: reduce churn at the application layer, then set the kernel values to absorb the residual and the bursts.

Safe order of operations and what not to do

Apply in this sequence, confirming each step with the counters: measure the four ceilings, widen the port range, enable tcp_tw_reuse, tighten keepalive to sit below your upstream idle timeout, raise file descriptors, and only then consider tcp_max_tw_buckets. Every value goes into /etc/sysctl.d/ with a comment explaining why, because a sysctl -w that is not persisted silently reverts at the next reboot and the failure returns weeks later with no obvious trigger.

Do not enable tcp_tw_recycle (removed, and previously broke clients behind NAT), do not set tcp_tw_reuse=2 on kernels older than 4.12, and do not chase TIME_WAIT with tcp_fin_timeout — it does not affect that state. If connection-heavy work has outgrown the instance even after all of this, the constraint is sizing versus isolation rather than sysctl values, and it is worth measuring CPU steal before buying a bigger plan.

Tuned this way, the 2 vCPU instance held 1,200 connections per second with zero errors and TIME_WAIT pinned under 10,000 — roughly seven times the headroom it had on stock settings.