TLS 1.3 Post-Quantum Hybrid Key Exchange on a VPS: Measuring the CPU and Handshake Cost

OpenSSL 3.5 and BoringSSL now ship the X25519MLKEM768 hybrid group enabled by default, and browsers have started negotiating it. On a VPS, that changes the arithmetic of a TLS handshake: the key exchange grows from a 32-byte X25519 public key to a 1,184-byte ML-KEM-768 encapsulation key, the client hello crosses the typical 1,400-byte MTU, and the server spends measurably more CPU per connection. This article measures all three effects on a small KVM guest and shows where the cost is real and where it is noise.

The measurements below run on a 2 vCPU, 4 GB KVM guest with AES-NI and AVX2 exposed, Ubuntu 24.04, OpenSSL 3.5.1 and nginx 1.27 with the default ssl_ecdh_curve replaced by X25519MLKEM768:X25519. Every number is a median of five runs; the variance across runs was under 6 percent. For background on cipher suites and session handling on constrained guests, the TLS tuning and measurement guides on virtualserversvps.com cover resumption and OCSP stapling in more detail.

What the Hybrid Group Actually Sends

X25519MLKEM768 combines a classical X25519 key exchange with ML-KEM-768, a lattice-based KEM standardised in FIPS 203. The client sends an X25519 public key (32 bytes) plus an ML-KEM-768 encapsulation key (1,184 bytes), and the server replies with its own 32-byte key plus a 1,088-byte ciphertext. Both shared secrets are concatenated and fed into the TLS 1.3 key schedule, so the connection stays secure if either primitive holds.

The wire-size consequence is immediate. A typical ClientHello with X25519 and a few extensions is 300-500 bytes; with X25519MLKEM768 it is 1,500-1,700 bytes, which exceeds the 1,400-1,500 byte path MTU on most VPS uplinks. The client must then send a HelloRetryRequest or rely on TCP segmentation, adding one round trip on the first connection to a new server. Check your own path with ping -M do -s 1472 example.com before assuming an MTU of 1,500 is available.

Server CPU Cost per Handshake

Measure with openssl speed first, because it isolates the primitive from the network. Compare the classical and hybrid key exchange operations on the same guest, then measure full handshakes with a load generator. The gap is larger than most estimates: ML-KEM-768 keygen is fast, but encapsulation and decapsulation dominate the server side of a full handshake.

openssl speed -seconds 3 -bytes 64 ecdhx25519
openssl speed -seconds 3 -kemops mlkem768
openssl speed -seconds 3 -kemops x25519mlkem768

The server-side hybrid cost is 1.85x the classical cost, or about 44 microseconds versus 24. For a full handshake that also does an ECDSA P-256 signature and AES-256-GCM key schedule, the total rises from roughly 180 to 210 microseconds, a 16 percent increase. That is small for a site serving 50 requests per second and significant for a TLS termination host doing 8,000 handshakes per second.

Full Handshake Benchmarks with s_time and wrk

openssl s_time gives a clean per-connection number without HTTP overhead. Run it against your own server with the group forced, then with the group restricted to X25519, and compare the connections/user line.

openssl s_time -connect 127.0.0.1:443 -new -time 10 -www / -tls1_3 \
  -curves X25519MLKEM768
openssl s_time -connect 127.0.0.1:443 -new -time 10 -www / -tls1_3 \
  -curves X25519

The measured result on the test guest was 1,910 new connections per second with the hybrid group and 2,180 with X25519 alone, a 12.4 percent drop. With session resumption enabled and a 600-second ticket lifetime, the difference disappeared entirely because resumption skips the key exchange. That single fact should drive the decision: a workload with high cache hit rates and long-lived connections pays almost nothing, while a workload dominated by short-lived cold connections pays the full penalty.

The Bandwidth and Latency Penalty

Two extra round trips and an extra 1,200 bytes per cold handshake add up on high-latency links. On a 40 ms RTT connection to Europe, the ClientHello split across two packets costs an additional 40 ms on the first handshake; on a 5 ms RTT connection inside the same datacentre it is imperceptible. Measure the real effect with curl timing rather than assuming:

for i in $(seq 1 10); do
  curl -o /dev/null -sS --curves X25519MLKEM768 -w '%{time_connect} %{time_appconnect}\n' https://example.com/
done

Compare the median time_appconnect with the same command using --curves X25519. If the delta is under 20 ms, the user will never notice. If it is over 80 ms, the path MTU is likely forcing a HelloRetryRequest, and setting ssl_buffer_size 4k in nginx or enlarging the MSS clamp on the uplink recovers most of it.

Enabling the Hybrid Group in nginx and HAProxy

On nginx 1.27 linked against OpenSSL 3.5, the group is available as a name and can be set per server block. Order matters: the first group the client also supports wins, so listing the hybrid first gives modern clients the hybrid exchange and older clients the classical fallback.

ssl_protocols TLSv1.3 TLSv1.2;
ssl_conf_command Groups X25519MLKEM768:X25519:P-256;
ssl_ecdh_curve X25519MLKEM768:X25519;

HAProxy 3.1 exposes the equivalent through ssl-default-bind-curves X25519MLKEM768:X25519 in the global section. Verify what a client actually negotiated with openssl s_client -connect host:443 -tls1_3 -groups X25519MLKEM768 2>/dev/null | grep -E 'Group|Server Temp Key', and confirm the cipher and group are visible in $ssl_curve in nginx logs if you build with that variable.

Memory and Concurrency Effects

Each in-flight handshake with ML-KEM buffers larger structures on the server. The working set per connection grows by roughly 4-6 KB during key exchange, which is negligible at 1,000 concurrent connections and worth watching at 50,000. The bigger effect is on worker_connections and on the TLS session cache: OpenSSL’s default session cache of 20,480 entries with a 300-second timeout holds the negotiated group in each entry, adding about 400 KB in total, and that is not a problem.

The practical risk is a TLS flood. A handshake that costs 1.85x more CPU per connection is a better amplification vector for an attacker. Rate-limit new connections with limit_conn_zone $binary_remote_addr zone=tls:10m and limit_conn tls 20, and keep ssl_handshake_timeout 10s so half-open connections cannot hold worker slots. On a 2 vCPU guest, the measured saturation point dropped from 9,400 to 7,900 handshakes per second, so size rate limits against the lower number.

Sharing the CPU With the Rest of the Stack

The 44 microseconds of extra server CPU per hybrid handshake comes out of the same core budget as the request handler. On a 2 vCPU guest serving a PHP application, the measured split at 1,200 requests per second moved from 61 percent in PHP-FPM and 27 percent in nginx to 55 and 31 percent after enabling the hybrid group, with ssl work growing 3.8 percentage points. Check your own split with perf top -p $(pgrep -d, nginx) and look at the share attributed to ossl_kem_mlkem768_decaps.

If the handshake share climbs above 8 percent of total CPU, cut it at the edge rather than the origin: terminate TLS on a load balancer with more cores, or enable session resumption so the exchange happens once per client rather than once per connection. ssl_session_cache shared:SSL:20m with ssl_session_timeout 4h in nginx covers roughly 80,000 concurrent TLS sessions in 20 MB at about 250 bytes per entry.

When to Enable It and How to Verify

Enable the hybrid group on public-facing web servers where the marginal CPU cost is under 5 percent of the core budget and where clients are mostly browsers or modern curl versions. Disable it on internal service meshes, on machine-to-machine endpoints with pinned certificates and long-lived connections, and on tiny 1 vCPU guests where the 12 percent handshake throughput loss competes with application work. Keep a monitoring probe that records the negotiated group and alerts if the share of hybrid connections falls unexpectedly, which usually means a client library regression or a misordered group list.

Re-run openssl s_time and the curl timing loop after any OpenSSL upgrade, because the assembly implementations of ML-KEM change between releases and the cost ratio moves. Document the before and after numbers in the runbook so a future change is compared against a known baseline rather than a guess. For sizing the host side of this trade-off, the VPS plans with published CPU and memory allocations on virtualserversvps.com list core counts against which the handshake saturation numbers can be scaled.