A PHP-FPM worker’s RSS goes from 48 MB to 140 MB over three days and nobody deployed anything. free -m shows plenty of available memory, yet latency climbs and the OOM killer eventually takes the worker. The explanation is usually in minor faults per second, not in RSS growth: an application allocating and freeing small chunks in a loop pays a page fault for every touched mapping, and each fault carries scheduler and TLB cost that top folds into softirq-adjacent noise. You can also compare the VPS plans behind these numbers before you commit.
Page faults are cheap individually and ruinous in aggregate. This is a measurement workflow that separates minor from major faults per process, attributes the cost to a specific PHP code path, and tells you whether the fix is allocator tuning, a memory limit, or a host-level RAM guarantee.
Minor versus major, and why the ratio is the signal
A minor fault maps a page that exists in the page cache or in the process’s anonymous memory but is not yet in the page table. It costs a trap, a zero or a copy of a 4 KiB frame, and entry into the fault handler—single-digit microseconds. A major fault has to reach block storage, which is tens to thousands of microseconds depending on whether the backing file is on NVMe or swap. The ratio between them tells you whether you have a mapping problem or a memory-pressure problem.
On a healthy PHP-FPM pool, minor faults run in the hundreds per second per worker while serving requests, and majors are effectively zero after opcache warms. Majors climbing steadily means pages are being reclaimed and re-faulted: swap thrashing, or an anonymous mapping that exceeds what the host has actually committed to you.
Per-process fault counters with pidstat
pidstat -r -p <pid> 5 prints minflt/s, majflt/s and VSZ/RSS per interval. Use the -C selector to watch every worker at once, and send the output to a file so you can compare intervals after the fact:
pidstat -r -C php-fpm8.3 5 120 > /var/log/faults.log
grep -v '^Average' /var/log/faults.log | awk '$4+$5>0'
Note that pidstat’s minflt counter can be affected by the kernel’s task-memory accounting granularity; the per-second rate is reliable, the absolute since-boot figure is not. Compare rates under identical request load: run the same ab or wrk pass during an idle period and during peak, and the delta in minflt/s attributable to concurrency becomes visible.
Watch the pool, not just one worker
PHP-FPM pre-forks, so a pool of 20 workers with 300 minflt/s each is 6,000 faults per second for the box—enough to consume a meaningful slice of a small vCPU. Sum across the pool with pidstat -r -C php-fpm8.3 1 1 | awk and divide by vCPU count to express the cost as a percentage of a core. Rough guidance: 10,000 minor faults per second costs roughly 3–5% of a modern core in fault entry alone, before the page-zeroing work.
Also check whether the faults are spread or concentrated. One worker with a high minflt rate and the rest quiet points at a specific endpoint or an extension; uniform rates across all workers point at the allocator, the opcache configuration, or a shared code path in every request.
Attribute the faults to a code path
perf record -e page-faults -g -p <pid> -- sleep 30 then perf report gives a call-graph view of who triggers the faults. For PHP this lands in zend_mm_alloc, mmap, or an extension’s buffer growth. Seeing faults concentrated in a single extension is actionable: it is usually a per-request allocation of a large temporary buffer that could be reused.
Where perf is unavailable, /proc/<pid>/stat fields 10 and 12 (minflt and majflt) sampled once a second gives the same rate without kernel tracing, and pairs well with the request log if you correlate by timestamp. A spike aligned with a specific route in the access log is the fastest attribution path there is.
Separating allocator churn from reclaim
If major faults appear only when RSS exceeds a threshold, and vmstat 1 shows si/so moving at the same moment, the process is being pushed into swap by pressure. Check /proc/<pid>/smaps_rollup for Swap: and compare MemAvailable in /proc/meminfo—a gap between what PHP-FPM reports as committed and what the host actually provides is the signature of a ballooning guest.
If RSS is stable, swaps are zero, and minors are high, the cause is allocator fragmentation: glibc’s per-thread arenas keep growing and returning pages to the kernel only under M_TRIM_THRESHOLD, so each request faults new pages that were never reused. Values from malloc-trim in the FPM pool config or switching to jemalloc change the fault rate measurably, and the measurement above is how you prove it rather than assume it.
Testing under controlled pressure
Reproduce deliberately: run a load test at normal concurrency, record minflt/s and majflt/s, then hold 85% of RAM with stress-ng --vm 1 --vm-bytes 85% --vm-keep and repeat. A pool that is healthy at baseline and shows double-digit major faults per second under pressure is one whose memory is not reserved—an allocation policy question rather than an application bug.
Transparent hugepages and the fault rate
THP changes the fault picture: a 2 MiB hugepage is faulted in as a single event, so minflt/s drops sharply while the cost per fault rises by an order of magnitude. A PHP worker that shows a sudden fall in minor faults after a kernel change has not become faster — it is faulting hugepages. grep AnonHugePages /proc/<pid>/smaps_rollup confirms whether the pool is using them, and /sys/kernel/mm/transparent_hugepage/enabled shows the policy.
Measure latency alongside the fault rate rather than instead of it. For a process doing fine-grained allocation, THP is a net loss because the zeroing cost of a 2 MiB page lands on the request that first touches it; for a process with large anonymous regions it removes millions of faults. The fault counters tell you which of the two you have, and the p99 tells you whether the trade was worth it.
That distinction decides who fixes it. Allocator and opcache changes are yours; guaranteed RAM is a plan question, and our VPS plan comparison is where the committed-versus-allocated difference is laid out. For pressure-driven fault behaviour that follows from how the kernel manages page cache, see the managed vs unmanaged breakdown. Either way, record the fault rates before and after each change — they move more reliably than wall-clock latency.




Leave a Reply
You must be logged in to post a comment.