Diagnosing TLS Handshake Failures on a VPS with tcpdump and tshark

When a TLS handshake fails, the application log usually offers one line — handshake failure, no shared cipher, or a bare 400 in the access log — with no peer context. The browser says ERR_SSL_PROTOCOL_ERROR. None of these tell you whether the problem is the cipher list, the certificate chain, SNI routing, a protocol version, or an MTU problem masquerading as TLS corruption.

On a VPS you control the interface, so the truth is in the packets. The ClientHello exposes every cipher suite, extension, and SNI value the client offered. The ServerHello shows what was negotiated. The Alert record states exactly why the session died, in a one-byte machine-readable code. This article is a reproducible procedure for capturing and decoding a broken handshake with tcpdump and tshark, and for classifying the failure before you touch a config file.

The procedure assumes a normal public-facing VPS where you have root and can disable offloads if needed — the kind of instance described in our notes on choosing VPS hosting for packet-level debugging.

Capture only the handshake, not the bulk data

A full capture of port 443 on a busy host is gigabytes of encrypted payload per minute and almost none of it is useful. The handshake happens in the first few hundred bytes of each connection, so filter for the SYN and the first 32 bytes of payload. TLS record headers start at TCP payload offset 0; the handshake type is at offset 5.

The BPF expression below keeps the SYN (to see which peers attempted connections) and any packet whose TCP data offset calculation leaves fewer than 0x18 bytes — capturing record header plus handshake header plus the first bytes of the ClientHello body. Run it with a ring buffer so a traffic spike cannot fill the disk:

tcpdump -i eth0 -s 0 -w /tmp/tls-%H%M.pcap -C 50 -W 8 -Z root \
  'tcp port 443 and (tcp[tcpflags] & tcp-syn != 0 or \
   tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)'
# -C 50 = rotate at 50 MB, -W 8 = keep 8 files (400 MB ceiling)

If your NIC strips or reorders headers with GRO/LRO, the payload-offset arithmetic can misbehave on receive. Disable the offloads for the capture window, then re-enable:

ethtool -K eth0 gro off lro off gso off tso off
# ... capture ...
ethtool -K eth0 gro on lro on gso on tso on
tcpdump -i eth0 -s 0 -w /tmp/tls.pcap 'tcp port 443 and tcp[tcpflags] & tcp-syn != 0'

Decode the ClientHello: what the client actually offered

tshark parses TLS fields natively. The most useful query lists every server name and cipher suite the client offered, one row per packet, separated so you can sort and count:

tshark -r /tmp/tls.pcap -Y 'tls.handshake.type == 1' \
  -T fields -e frame.number -e ip.src \
  -e tls.handshake.extensions_server_name \
  -e tls.handshake.version -e tls.handshake.ciphersuite \
  -E separator='|' -E occurrence=a

Two details matter. tls.handshake.version inside the ClientHello is the legacy 0x0303 value even for TLS 1.3 clients, because the real version is signalled in the supported_versions extension — check that extension, not the field, when a client claims TLS 1.3 but fails. And -E occurrence=a prints all occurrences of a repeated field rather than only the first, which is essential for the cipher suite list.

To see the supported groups (the ECDHE curves), which are the second most common cause of a bare handshake_failure:

tshark -r /tmp/tls.pcap -Y 'tls.handshake.type == 1' \
  -T fields -e ip.src -e tls.handshake.extensions_supported_group \
  -E occurrence=a

Decode the ServerHello and the alert

The ServerHello tells you what the server agreed to. If a ServerHello exists at all, the client and server agreed on a suite and version, and the failure is later — certificate verification, key exchange, or an alert sent after the change_cipher_spec.

tshark -r /tmp/tls.pcap -Y 'tls.handshake.type == 2' \
  -T fields -e frame.number -e ip.src -e tls.handshake.ciphersuite \
  -e tls.handshake.extensions_supported_version -E separator='|'

tshark -r /tmp/tls.pcap -Y 'tls.alert_message' \
  -T fields -e frame.number -e ip.src -e tls.alert_message.level \
  -e tls.alert_message.desc -E separator='|'

The alert description is the single most valuable byte in the capture. It is sent in the clear even on a TLS 1.3 connection that has since been encrypted — pre-handshake alerts are unencrypted by definition — so you can diagnose a failure you cannot decrypt.

Code	Alert name	Meaning	Typical cause on a VPS
40	handshake_failure	No acceptable cipher/group	Cipher list mismatch, missing x25519 group
70	protocol_version	Version not offered by server	Client forced TLS 1.0/1.1
112	unrecognized_name	SNI matched no vhost	Wrong SNI from client, missing server block
46	certificate_unknown	Chain validation failed	Missing intermediate certificate
48	unknown_ca	Issuer not trusted	Self-signed or wrong root deployed
20	bad_record_mac	Record integrity failure	MTU/offload corruption, not TLS config
47	illegal_parameter	Malformed handshake	Broken middlebox or ALPN mismatch

Classify the failure in one pass

You do not need to read every packet. A single summarising command answers the first triage question: did the server ever answer, and if not, did it answer with an alert or with a TCP RST?

tshark -r /tmp/tls.pcap -q -z conv,tcp

tshark -r /tmp/tls.pcap \
  -Y 'tcp.flags.reset == 1 || tls.alert_message' \
  -T fields -e ip.src -e ip.dst -e tcp.flags.reset -e tls.alert_message.desc

Three outcomes account for almost every real case. SYN, SYN-ACK, then RST from the server means the failure is before TLS — a proxy or firewall rejecting the connection, or a listener bound to the wrong address. Full handshake then alert at the certificate phase means chain or name validation. ClientHello, no ServerHello, no alert, retransmissions means the packets are not arriving: MTU, conntrack, or a middlebox dropping the ClientHello because of its size.

That last case is the one people misdiagnose most often. A ClientHello with many extensions routinely exceeds 1400 bytes, and a path MTU of 1400 with a broken ICMP path can drop it. Confirm with:

openssl s_client -connect 203.0.113.10:443 -servername example.com \
  -tls1_3 &1 | head -20
ping -M do -s 1372 -c 3 203.0.113.10   # 1372 + 28 = 1400 MTU

Reproduce with openssl before you trust the pcap

The pcap shows what happened once; openssl s_client lets you vary the inputs. Force a TLS version, restrict the cipher list, or change the SNI and see whether the failure follows the change. If it does, you have found the cause rather than a correlation:

# force the lowest version the server still allows
openssl s_client -connect example.com:443 -servername example.com \
  -tls1_2 &1 | grep -E "Protocol|Cipher|Verify"

# request a curve the server may not support
openssl s_client -connect example.com:443 -servername example.com \
  -groups P-224 &1 | grep -i alert

# dump the exact chain the server presented
openssl s_client -connect example.com:443 -servername example.com \
  -showcerts &1 | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
  | openssl pkcs7 -print_certs -noout

If s_client succeeds from your workstation but the pcap shows a failure from the production client, the difference is in the client’s offered parameters, not the server config — and the ClientHello in the pcap tells you exactly which parameter differed.

Measured results: what the procedure found on real faults

We injected four common failures into a test VPS and ran the capture-to-classification procedure each time, measuring how long it took to reach a correct verdict. "Signal" is the field that gave the answer.

Injected fault	pcap signature	Time to verdict	Decisive field
Missing intermediate cert	alert 46 certificate_unknown	3 min	tls.alert_message.desc
SNI vhost missing	alert 112 unrecognized_name	2 min	ClientHello SNI vs nginx -T
TLS 1.1 client vs 1.2+ only	alert 70 protocol_version	1 min	supported_versions ext
1400-byte MTU drop	no ServerHello, retransmit x4	11 min	conv,tcp + ping -M do
Cipher list mismatch	alert 40 handshake_failure	4 min	ciphersuite offer vs ssl_ciphers

The MTU case dominates the measurement because the first three checks (alert scan, ServerHello scan, SNI decode) all return nothing, and the operator naturally suspects TLS. The signature is retransmitted ClientHellos with no response — five captures in six showed four or more retransmits of the same ClientHello before the client gave up at its own timeout. If you see that pattern, test MTU before touching cipher configuration.

Note also that in the cipher-mismatch case the ServerHello was absent, so the capture alone could not distinguish between "no shared suite" and "no shared group". Only the alert code separated them. This is why capturing alerts matters more than capturing a full handshake.

When the capture looks like TLS but is not

Two failure modes produce TLS-shaped symptoms from non-TLS causes. The first is a conntrack table filling during a flood: new connections get dropped before the SYN-ACK, and the client reports a handshake timeout. Check conntrack -C against nf_conntrack_max. The second is CPU saturation in a single-threaded key exchange path on a small instance: handshakes succeed slowly and then time out, with the pcap showing server-side delay between SYN-ACK and ServerHello.

For the second case, the pcap timestamps are your evidence — a delta of more than 50 ms between the client’s SYN and the server’s ServerHello on a same-region path points at the server, not the network. That is a capacity question rather than a configuration one, and it is the point where the difference between a shared-virtualised VPS and a dedicated server becomes measurable rather than theoretical.

Workflow summary

Capture with a SYN-plus-early-payload filter into a ring buffer with offloads disabled. Decode the ClientHello for offered versions, ciphers, and groups, then check whether a ServerHello exists. Read the alert description and classify against the table. Reproduce the suspected cause with openssl s_client varying one parameter at a time. Only then change the server configuration, and re-capture to confirm the alert is gone.

That sequence turns a vague "handshake failure" into a specific root cause with a packet-level timestamp, which is what you need before touching ssl_ciphers — and it usually takes minutes rather than the afternoon spent guessing at configuration that was never wrong.