Measuring Page-Cache Hit Ratio on a VPS with vmtouch and cachestat

free -h showed 412 MB of 3.8 GB “available” on a web node also holding 610 MB of page cache for static assets — and could not say whether that cache was helping. Two tools answer it: vmtouch for which files are resident, cachestat for what fraction of reads the page cache serves. Together they turn a memory graph into a hit ratio with a target.

This walkthrough uses Debian 12, kernel 6.1, and a 3.8 GB VPS serving nginx from a 2.1 GB asset tree. Everything below is a command you can run on a live box without restarting anything.

Why /proc/meminfo cannot give you a hit ratio

MemAvailable, Cached and Buffers are instantaneous snapshots, not rates. Cached rising tells you the cache is growing, not whether it is serving reads. The kernel does expose aggregate counters in /proc/vmstat — pgpgin, pgpgout, pgmajfault — but they are cumulative since boot and mix file-backed and anonymous pages, which makes rate computation awkward and misleading.

grep -E "pgpgin|pgmajfault|nr_file_pages" /proc/vmstat
free -m
# MemAvailable is an estimate; it cannot say which files are cached

A page-cache hit costs roughly 100 ns of copy; a miss costs an 80 µs NVMe read. At 20,000 file reads per second a 10% shift in hit ratio means 2,000 extra device reads per second — idle disk versus saturated.

cachestat: hit ratio as a rate

cachestat ships with bcc-tools and reads the kernel’s per-CPU cachestat counters in rate form. It prints hits/misses/dirty per one-second window; the ratio is hits/(hits+misses).

# apt install bcc-tools  (or bpfcc-tools on Debian)
cachestat 1

# example output
#     HITS   MISSES  DIRTIES  HITRATIO   BUFFERS_MB  CACHED_MB
#   142030     3120     1801    97.8%          192      3184
#   138911     4102     2043    97.1%          192      3211

A cold cache after echo 3 > /proc/sys/vm/drop_caches started at 21.4% and settled at 97.6–98.2%. The interesting figure is the 34 seconds to reach 95% — how long a reloaded worker serves slower responses. Never drop caches on production to “test performance”: it is a self-inflicted latency incident with a measurable duration.

Run it under real traffic: on an idle box the ratio is dominated by systemd and cron reads. A useful run is 60 seconds at normal request rate, compared between configurations rather than trusted absolutely.

vmtouch: which files are resident, and how much RAM they need

vmtouch queries the page cache directly via mincore(2) per page. On a directory it reports how much of the tree is resident and, with -v, which individual files are fully or partly cached.

apt install vmtouch
vmtouch /var/www/static          # summary: resident vs total
vmtouch -v /var/www/static | head -20   # per-file residency
vmtouch -l /var/www/static       # lock into RAM (needs CAP_IPC_LOCK)

After an hour of traffic vmtouch reported 1,842 of 2,151 MB resident (85.6%), the missing 309 MB concentrated in 41 rarely-requested files — exactly what LRU should evict. That is more actionable than any global Cached figure.

vmtouch -l pins page cache the kernel cannot reclaim. On this 3.8 GB box, locking 2.1 GB left 1.3 GB for the application and produced an OOM kill within four hours. Use -t to preload without pinning, or keep the lock under 25% of RAM.

Computing a hit ratio you can track over time

cachestat gives one-second samples; to track a ratio over days, run it under a wrapper that emits a line per interval to your metrics agent, or sample /proc/vmstat pgpgin deltas against application read counts.

# 60-second mean hit ratio, parsed from cachestat
cachestat 1 60 | awk 'NR>2 {gsub(/%/,"",$4); h+=$4; n++} END {printf "mean hit ratio: %.2f%% over %d samples\n", h/n, n}'

# per-process file read hit proxy via BPF (requires root)
vfsstat 5 1

vfsstat 5 1 from bcc-tools breaks reads/writes down by operation; a rising Read rate with a flat Cached growth is the signature of a working set that has outgrown RAM. Pair it with nstat -az | grep -i drop if the source is a network mount rather than local disk.

Reading the numbers: what a good ratio looks like

There is no universal target, but the shape of the distribution is informative. Below are the same box under three traffic levels, with the working set fixed at 2.1 GB and RAM at 3.8 GB.

Traffic	cachestat hit ratio	vmtouch resident	Device read IOPS	p99 latency
20 req/s	98.1%	2,148 / 2,151 MB	38	9.4 ms
200 req/s	97.4%	2,149 / 2,151 MB	410	11.2 ms
800 req/s	94.8%	2,151 / 2,151 MB	1,720	19.8 ms
800 req/s + 900 MB DB cache	71.3%	1,540 / 2,151 MB	9,300	63.0 ms

The last row is the failure mode worth memorising: adding a database to the same 3.8 GB box pushed the asset working set out of cache, the resident fraction fell from 100% to 71.6%, and device read IOPS rose 5.4×. Nothing in free -h flagged it — available looked the same, because the pages had been reused rather than freed. Only the hit ratio and vmtouch residency showed the cause.

A ratio in the mid-90s with residency near 100% means more RAM buys nothing. A ratio below 90% with residency under 85% means the working set does not fit: shrink it or buy memory. In between, vmtouch -v shows whether a few large files crowd out many small ones.

Practical fixes when the ratio is low

• Shrink the working set. Recompressing JPEGs and PNGs to WebP/AVIF on this tree cut it from 2.1 GB to 940 MB, moving it comfortably inside RAM and taking the hit ratio to 99.4% under the 800 req/s load.
• Separate caches by workload. A database buffer pool and a file cache compete for the same pages; on a box that must host both, cap one with cgroup memory.high so it cannot evict the other’s working set.
• Use noatime. On this box atime updates were generating metadata writes that forced pages out prematurely; mounting with noatime raised the steady-state ratio by 1.3 points.
• Preload after deploys. vmtouch -t /var/www/static reads the tree into cache sequentially in about 2.4 seconds at 900 MB/s, avoiding the 34-second re-warm window.

Putting it in a check

A five-minute check catches the regression before users do: emit the ratio and alert on a sustained drop below baseline minus five points.

#!/bin/sh
# /usr/local/bin/cache-ratio.sh
cachestat 1 5 | awk 'NR>2 {gsub(/%/,"",$4); s+=$4; n++} END {printf "%.2f\n", s/n}'

# alert if below 90
ratio=$(/usr/local/bin/cache-ratio.sh)
awk -v r="$ratio" 'BEGIN {exit !(r < 90)}' && \
  logger -t cachecheck "page-cache hit ratio ${ratio}% below threshold"

Keep the threshold in monitoring, not the script; the shell version makes the measurement reproducible on any host without a full agent. For how much RAM should go to cache versus application heap, see virtualserversvps.com.