Transparent Huge Pages on a VPS: Measuring THP Effects on Latency and Memory

Transparent Huge Pages let the kernel back memory with 2 MB pages instead of 4 KB pages, cutting TLB misses and page-table walk depth. On bare metal this is usually a small win. On a virtualized guest it is a different calculation: 2 MB pages map to the hypervisor’s memory management in ways that can add swap-in cost, extend allocation stalls, and — under the wrong mode — turn a routine malloc into a millisecond-scale event.

The failure mode is subtle. THP does not usually make throughput worse; it makes the tail of the latency distribution worse. Median request times stay flat while p99.9 doubles, and the symptom looks like a database tuning problem rather than a page-size problem.

The three THP modes and what each actually does

THP is controlled per-system through /sys/kernel/mm/transparent_hugepage/enabled, which accepts three values: always, madvise, and never.

cat /sys/kernel/mm/transparent_hugepage/enabled
# [always] madvise never
cat /sys/kernel/mm/transparent_hugepage/defrag
# [always] defer defer+madvise madvise never

In always mode, any anonymous mapping large enough can be backed by a 2 MB page the moment it is faulted in, with no cooperation from the application. In madvise mode, allocation only happens where the program calls madvise(MADV_HUGEPAGE) on a range — the default for most modern allocators is to ask for it only on large arenas. In never mode, THP is off entirely and every page is 4 KB.

The second file matters as much as the first. defrag governs what the kernel does when a 2 MB page cannot be formed immediately. Under always, direct compaction runs synchronously in the faulting thread, which is exactly the multi-millisecond stall that shows up as a latency spike.

ModeWho requests THPTypical effectBest fit
alwaysKernel, automaticallyHigher throughput, longer tail latenciesBatch analytics, stateless compute
madviseApplication explicitlyOpt-in wins on hot mappings onlyMixed workloads, most web stacks
neverNobodyPredictable 4 KB behaviourDatabases with tight fsync SLAs

khugepaged: the background collapse

Even in madvise mode, khugepaged scans memory and collapses contiguous 4 KB pages into 2 MB pages opportunistically. It runs with a target scan rate (pages_to_scan, default 4096) and sleep interval (alloc_sleep_millisecs, default 60000). The work is done on behalf of whichever process owns the pages, so the cost lands on your application, not on a neutral background thread.

ps -o pid,comm,pri,ni -C khugepaged
grep -E 'pages_to_scan|alloc_sleep|scan_sleep' \
  /sys/kernel/mm/transparent_hugepage/khugepaged/*
# collapse count since boot
grep -E 'thp_collapse_alloc|thp_fault_alloc' /proc/vmstat

The two vmstat counters are the clearest signal of whether THP is actually doing anything. thp_fault_alloc counts huge pages allocated on fault; thp_collapse_alloc counts ones produced by khugepaged. If both are flat while your RSS grows, THP is effectively dormant and it is not the cause of your problem.

Measuring the effect on tail latency

You cannot judge THP from averages. Run a fixed workload under each mode and compare percentiles. A repeatable procedure on a VPS with PostgreSQL or MySQL works like this: pin to a quiet period, run the same read benchmark three times per mode, and record p50, p99, and p99.9.

# per mode: echo madvise > .../enabled ; then
sysbench oltp_read_only --threads=8 --time=120 \
  --rand-type=uniform --report-interval=10 run | tee out.$MODE

# page-fault and TLB cost attribution
perf stat -e dTLB-load-misses,dTLB-store-misses,\
page-faults,minor-faults -p $(pgrep -n mysqld) sleep 30

Two patterns are typical. With always, dTLB-load-misses drops by 60–90% and it is easy to conclude THP is free. But perf record -e kmem:mm_page_alloc -g during the same run will often show compaction and allocation stalls inside worker threads, and that is where the p99 regression comes from. With never, TLB misses rise but the distribution tightens. Which trade you want depends on whether you are optimized for throughput or for a latency promise.

Why VPS guests behave differently from bare metal

Three virtualization-specific effects change the calculus. First, guest physical memory is host virtual memory; a 2 MB guest page is not necessarily backed by a 2 MB host page, and the mapping work at the hypervisor layer does not shrink just because your guest page is bigger. Second, ballooning and KSM interact badly with huge pages: KSM cannot deduplicate a 2 MB page that differs in one byte, so THP reduces the dedup ratio on hosts that rely on it. Third, memory overcommit on the host means a 2 MB page fault may trigger host-level reclaim — cost that appears to you as unexplained stall time with no local cause.

Practical consequence: on a VPS, the marginal throughput benefit of THP is smaller than on bare metal, while the tail-latency risk is similar or larger. That shifts the default toward madvise, and toward never for fsync-bound databases.

If you are choosing a plan for a latency-sensitive database, the VPS configurations on virtualserversvps.com show memory and storage per tier, which is the relevant input: THP tuning buys you a few percent, but undersized RAM forces you into the swap path where huge pages stop helping entirely.

A sane default and how to apply it

Unless you have measured a benefit from always, run madvise with defrag set to defer so that compaction never runs synchronously in the faulting thread. Persist it with a systemd unit rather than a one-off echo, because sysfs settings do not survive reboots:

# /etc/systemd/system/thp-tune.service
[Unit]
Description=Set THP policy
After=sysinit.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled; echo defer > /sys/kernel/mm/transparent_hugepage/defrag'

[Install]
WantedBy=multi-user.target

For a database that owns its own buffer pool and has already sized it to fit in RAM, never is defensible and simpler to reason about — the buffer pool does not need TLB coverage because it is accessed through a small number of hot pages. For a JVM or Go service with a large heap, madvise plus checking whether the runtime already issues MADV_HUGEPAGE is the better starting point.

Then re-measure. THP is one of the few kernel knobs where the correct answer is site-specific rather than universal, and the p99.9 number from your own workload is the only evidence that should settle it.

Leave a Reply