MTU and PMTUD Black Holes: Diagnosing the Stalls That Look Like TLS Failures

Most TLS failures that are not certificate errors are MTU problems wearing a disguise. The TCP handshake completes, the certificate verifies, the first small HTTP response arrives — and then the transfer hangs the moment a large record is sent. The cause is Path MTU Discovery failing somewhere between client and server, so the server keeps transmitting full-size segments that are silently dropped, retried, and dropped again. This guide shows how to confirm a PMTUD black hole with a two-minute test, how to size MSS clamping on a tunnel or overlay, and how to verify the fix with tcpdump and per-socket counters. It assumes a Linux instance, run on a provider network where you control the guest but not every hop.

The Failure Signature

Recognise it by shape rather than by any single error message:

• Small responses succeed (curl -I works), large ones stall or take exactly 30 seconds — the kernel’s MSS retry ladder.
• The connection establishes, TLS handshake completes with a small certificate chain, then the body never arrives.
• ss -ti shows a socket with a small cwnd, non-zero retransmits and unacked stuck at 1–3.
• The same client works fine to other servers, and the problem correlates with VPN, PPPoE, or a mobile carrier.

<code>ss -tin state established '( dport = :443 )'
# estab 0 0 10.0.0.5:443 198.51.100.20:51234
#  cubic wscale:7,7 rto:204 rtt:28.1/4.2 ato:40 mss:1448 pmtu:1500
#  retrans:0/14 rcv_mss:1400 unacked:2</code>

retrans:0/14 on a fresh connection is the tell. Fourteen retransmits of the same lost segment means the segment is being dropped downstream and no ICMP “fragmentation needed” is coming back to tell the sender to shrink.

Test 1: Confirm PMTUD Fails From the Server Side

Send packets of increasing size with the Don’t Fragment bit set and see where they stop getting through. ping with -M do does exactly this:

<code>#!/bin/sh
# sweep DF-set payloads from 1200 to 1472 bytes
for s in 1200 1300 1372 1400 1442 1452 1472; do
  printf '%-6s ' "$s"
  ping -c 2 -W 1 -M do -s $s 198.51.100.20 \
    | grep -q '1 received\|2 received' && echo OK || echo FAIL
done</code>

The largest successful payload plus 28 bytes of IPv4 and ICMP headers is the path MTU. If 1400 succeeds and 1442 fails, the path MTU is between 1428 and 1470 — very likely 1420, a common PPPoE or WireGuard value. Check what the kernel learned:

<code>ip route get 198.51.100.20
ip route show cache | grep -i mtu
cat /proc/sys/net/ipv4/route/mtu_expires   # default 600s</code>

A cached mtu lower than the interface MTU means PMTUD is working for that destination — the problem is elsewhere. No cached MTU combined with the test failures above means ICMP is filtered.

Why ICMP Filtering Breaks Everything

Path MTU Discovery relies on routers sending ICMP type 3 code 4 (fragmentation needed) back to the sender. Firewalls treat ICMP as noise and drop it. The classic mistake is one line in an nftables or iptables ruleset: ct state established,related accept followed by policy drop, with no rule permitting ICMP.

<code># nftables: permit the ICMP types PMTUD and error reporting depend on
nft add rule inet filter input ip protocol icmp icmp type \
  { destination-unreachable, time-exceeded, parameter-problem } accept
nft add rule inet filter input ip6 nexthdr icmpv6 \
  icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem } accept</code>

For IPv6, dropping packet-too-big is catastrophic because IPv6 routers never fragment — PMTUD is mandatory, and without it every connection above roughly 1280 bytes stalls. Rate-limit ICMP rather than dropping it:

<code>nft add rule inet filter input icmp type destination-unreachable \
  limit rate 20/second burst 40 packets counter accept</code>

Test 2: Trace the Black Hole

When the sweep fails, capture what actually happens during a failing transfer. Watch for duplicated sequence numbers with full-size payloads and no ICMP arriving:

<code>tcpdump -ni eth0 -s0 'host 198.51.100.20 and tcp port 443' -c 200 -w /tmp/pmtud.pcap
tcpdump -nr /tmp/pmtud.pcap 'tcp[tcpflags] & tcp-syn == 0' | head -20</code>

The signature is a repeated segment of length 1448 followed by a retransmission of the same sequence, with no smaller segment in between until the RTO ladder finishes. If a smaller segment does appear after an ICMP arrives, PMTUD is functioning and the stall has a different cause — an application-level timeout, or a load balancer with a mismatched idle timeout.

For a documented example inside the kernel, enable TCP metrics export:

<code>ss -tin -o state established '( dport = :443 )' | grep -E 'retrans|mss|cwnd'
nstat -az TcpExtTCPMTUPFail TcpExtTCPMTUPSuccess TcpRetransSegs</code>

TcpExtTCPMTUPFail incrementing is direct evidence that MTU probing timed out on that route.

MSS Clamping: The Server-Side Fix

You cannot fix the client’s broken network, but you can stop the server from sending oversized segments. MSS clamping rewrites the MSS option during the handshake so both ends agree on a segment size that fits the smallest link:

<code># clamp only on the interface facing clients
nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu
# or force a fixed value (e.g. 1360 for a 1400-byte path)
nft add rule inet filter forward tcp flags syn tcp option maxseg size set 1360

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu</code>

With rt mtu the kernel uses the route MTU at handshake time, which is correct on a tunnel whose MTU is configured properly. The fixed-value form is for paths you know to be narrow regardless of routing — a WireGuard overlay at 1420 bytes wants MSS 1380 at most:

Interface MTU and Kernel Behaviour

Lower the interface MTU when the whole path is narrower than the configured value — this is mandatory on tunnels:

<code>ip link set dev wg0 mtu 1420
ip link set dev eth0 mtu 1500
cat /proc/sys/net/ipv4/tcp_mtu_probing
cat /proc/sys/net/ipv4/tcp_base_mss</code>

tcp_mtu_probing is the safety net for clients behind a black hole: with value 1 the kernel enables probing only after an initial connection stalls, and with 2 it probes always. Setting it to 2 on a public-facing server costs a few extra round trips per connection but makes large transfers survive broken paths:

<code>sysctl -w net.ipv4.tcp_mtu_probing=1
sysctl -w net.ipv4.tcp_base_mss=1024
echo 'net.ipv4.tcp_mtu_probing=1' >> /etc/sysctl.d/99-net.conf</code>

Do not enable probing globally as a substitute for fixing ICMP filtering. It masks one specific failure, adds latency, and leaves UDP-based protocols — QUIC included — unprotected.

Verification Checklist

1. DF sweep from server to client identifies the real path MTU per destination.
2. ICMP types 3 and 11 (and ICMPv6 packet-too-big) permitted inbound, rate-limited.
3. Interface MTU correct on every tunnel, matching the inner encapsulation overhead.
4. MSS clamping applied to SYN packets on the client-facing path.
5. ss -tin shows no persistent unacked segments and a plausible pmtu value.
6. Large transfer completes in a time consistent with bandwidth-delay product, not the 30-second RTO ladder.

Run the sweep from at least two networks — a datacenter host and a consumer connection — because a black hole is often visible only from one side. Once the path MTU is known and clamping is in place, the 30-second stalls disappear without any change to TLS configuration: nothing was wrong with the certificate path in the first place. If large transfers are still slow after the fix, the limit has moved to socket buffers, and the underlying instance’s bandwidth and buffer allocation is the next thing to measure rather than another MTU experiment.