cgroup v2 Memory High and Watermark Protection: Tuning Real Workloads on a VPS

Most guides treat cgroup memory control as one knob: set memory.max and let the OOM killer sort it out. cgroup v2 exposes four controls with different semantics, and the most valuable — memory.high — kills nothing. It throttles the cgroup into direct reclaim, forcing it to give back page cache, and escalates only if the cgroup cannot comply. On a 2-4 GB VPS running three or four services, that is the difference between one slow service and the whole machine dying.

The four memory controls and what each enforces

cgroup v2 memory controller files live in the cgroup directory, typically /sys/fs/cgroup/<unit>/ under systemd. Each has distinct behaviour:

memory.max is the wall, memory.high the speed bump before it, memory.low a suggestion to the reclaim algorithm, and memory.min a real reservation. Most incidents on small VPS instances come from setting only memory.max and never reclaiming the 400 MB of cache the application was holding.

How throttling actually happens under memory.high

When a cgroup’s charge crosses memory.high, each subsequent allocation must trigger reclaim proportional to how far over it is. The kernel does not run the OOM killer; it makes the allocating task reclaim synchronously, so the application stalls inside malloc or a page fault. Reclaim order matters: clean page cache first, then dirty pages written back and dropped, then file-backed executable pages, then anonymous pages pushed to swap. If swap is unavailable — common on small plans — anonymous memory cannot be reclaimed and the cgroup can only shrink its cache. Once there is nothing left, the next allocation hits memory.max.

cat /sys/fs/cgroup/myservice/memory.current
cat /sys/fs/cgroup/myservice/memory.high
cat /sys/fs/cgroup/myservice/memory.events
# memory.events shows: low high max oom oom_kill oom_group_kill

memory.events is the audit trail. Rising high with stable oom_kill means throttling works. max incrementing means you hit the wall and reclaim failed; oom_kill incrementing means something died.

Page cache reclaim is the point, not a side effect

On a 2 GB box, a Node.js or Java service that reads many files accumulates hundreds of megabytes of page cache. That cache counts in memory.current but is free memory — dropping it costs a re-read. Setting memory.high at 85-90 percent of memory.max makes the service eat its own cache before the system comes under pressure.

Without memory.high, global pressure reclaims proportionally across cgroups, so your database’s hot pages get dropped to make room for a build job. memory.low biases reclaim away from the database but only as a weight; memory.min is the only guarantee.

# systemd drop-in for a 2 GB box, service capped at 1.2 GB
[Service]
MemoryMax=1200M
MemoryHigh=1000M
MemoryLow=400M
MemorySwapMax=0

Apply with systemctl daemon-reload and verify with systemctl show myservice -p MemoryMax -p MemoryHigh; cat memory.high should return 1048576000 for 1000M.

Practical thresholds for a 2-4 GB VPS

Start from resident working sets, not total RAM. Measure each service’s RSS under realistic load with smem -tk, then add 15-25 percent for fragmentation and transient buffers.

• 2 GB box, single app: MemoryHigh 1.4G, MemoryMax 1.6G, leaving 300-400 MB for the kernel, journald and SSH.
• 2 GB box with MySQL 512 MB + app: MySQL MemoryLow 400M / Max 600M; app High 700M / Max 800M.
• 4 GB box, three services: each High at 80 percent of its Max, and the sum of Max values not exceeding 3.4 GB.
• Swap present: set MemorySwapMax to 0 for latency-sensitive services so throttle happens before swap thrash.

Never let the sum of memory.max equal total RAM: the kernel, page tables and slab need 8-12 percent, and systemd’s slices take a little too. On a single-service box at the entry VPS tier the exercise often collapses to one number — memory.high at MemTotal minus 400 MB — so the throttle fires before any host-level balloon or global OOM.

Verifying with memory.pressure

memory.pressure reports the share of wall-clock time tasks were stalled on memory. some counts any stalled task, full counts periods where every runnable task was stalled — the signal for whether the throttle worked or just moved the stall elsewhere.

cat /sys/fs/cgroup/myservice/memory.pressure
# some avg10=12.40 avg60=9.10 avg300=4.20 total=88123456
while true; do
  printf '%s ' "$(date +%T)"
  grep -o 'full avg10=[0-9.]*' /sys/fs/cgroup/myservice/memory.pressure
  sleep 5
done

Interpretation for a 2-4 GB box:

• full avg10 under 2 percent: healthy, the cgroup has headroom.
• full avg10 between 2 and 10 percent: throttle is active; check whether latency matters and raise memory.high if p99 user latency is affected.
• full avg10 above 20 percent sustained: the working set does not fit; reduce it or resize the plan.
• Zero pressure but rising memory.events:max: allocations are spiking past the wall between samples; lower memory.high so throttling engages earlier.

Pair it with /proc/pressure/memory: high host-wide pressure alongside low cgroup pressure means the contention is other tenants or a host balloon.

A tuning sequence that converges in three passes

Apply this in order — each pass depends on the observation from the previous one.

# pass 1: observe
systemd-cgtop -m --iterations=12
cat /sys/fs/cgroup/myservice/memory.peak
# pass 2: throttle above observed peak
echo 1100M > /sys/fs/cgroup/myservice/memory.high
# pass 3: hard ceiling with margin
echo 1250M > /sys/fs/cgroup/myservice/memory.max
systemctl restart myservice && sleep 300
grep -E 'high|max|oom' /sys/fs/cgroup/myservice/memory.events

memory.peak gives the high-water mark since creation — size memory.high from that, about 10 percent above, not from the instantaneous reading. After a week, if high is in the millions and oom_kill is zero, pressure is being absorbed as latency instead of death. If max is climbing, the ceiling is one spike from an OOM. Recording both counters next to your VPS plan’s memory allocation makes the next resize decision evidence-based.