Why a Process’s RSS Grows for Hours Without a Leak

A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind reports nothing, gdb finds no growing references, and heaptrack shows total allocations flat. The process is not leaking. It is fragmenting, and the allocator is hoarding pages that it has already freed from your point of view.

This pattern accounts for many “mystery OOM” incidents on long-lived services in memory-limited cgroups. glibc’s malloc does not give memory back eagerly; it keeps freed chunks ready for reuse, because reuse is cheaper than a syscall. The trade is RSS that only rises until something forces a trim — often the OOM killer.

The fixes have side effects, so understanding the mechanism matters. A new arena per thread can multiply RSS by 8× on a 64-thread process; dropping MALLOC_ARENA_MAX to 1 can serialise a previously lock-free workload. When the memory cgroup limit is the real ceiling, both mistakes are fatal in opposite directions.

How glibc malloc actually organises memory

Below 128 KB (the M_MMAP_THRESHOLD default), an allocation comes from an arena: a 64 MB reserved region grown with brk for the main arena and mmap for thread arenas. Inside an arena, free chunks of different sizes coalesce only with adjacent neighbours; a 40-byte chunk wedged between two live 1 MB chunks fragments that 64 MB region for the rest of the process’s life. The arena cannot reclaim the middle.

Then there is the second layer: only the top chunk of an arena can be returned with brk, and only if it exceeds M_TRIM_THRESHOLD (default 128 KB). A process that allocates and frees in a pattern that leaves a live chunk near the top will never satisfy that condition. This is why RSS grows in steps and never comes back down: the arena grew, the top chunk is pinned, and every subsequent allocation reuses the holes first.

# see arenas and fragmentation at runtime
MALLOC_ARENA_MAX=4 MALLOC_STATS=1 ./svc
mallinfo2()  # uordblks vs fordblks vs hblkhd
cat /proc/$PID/smaps_rollup | egrep 'Rss|Pss'

# force a trim without restarting
python3 -c "import ctypes;ctypes.CDLL('libc.so.6').malloc_trim(0)"

Why threads multiply the problem

Each thread that allocates and finds the main arena contended gets its own arena, up to 8 × nproc on 64-bit. A 32-thread service on a 16-core host can end up with 32 arenas, each of which may grow to several hundred MB under bursty allocation. mallinfo2().arena divided by 1024 and summed per arena shows the total; if that number tracks RSS closely, fragmentation is the driver.

Setting MALLOC_ARENA_MAX=4 caps the count and is usually the highest-value single change. The cost is lock contention: measure throughput before and after. Most request-serving services lose under 3%; a parser doing millions of small allocations per second can lose 20%, in which case jemalloc with per-thread caches is the better answer.

free() returns to the arena, not to the kernel

When free() is called, the chunk goes back to the arena’s free list. The kernel is not involved. The page backing that chunk is not madvised away unless MADV_DONTNEED trimming triggers, which happens only at the top of the arena or during an explicit malloc_trim. So free() of a 900 MB buffer of 64 KB chunks can leave RSS unchanged, and top will report the process as still holding that memory.

This is correct behaviour for a general-purpose allocator optimising for speed, and wrong for a container that is billed by RSS. It also explains why RSS is a lagging indicator: tools like ps show pages the allocator owns, not bytes your code can reach. The gap between those two numbers is the fragmentation you need to measure, and uordblks versus arena from mallinfo2 gives it to you directly.

When to trust heap profiling, and when not to

Heap profilers attribute live allocations to call sites. They are excellent at finding a genuinely retained object graph and useless at finding fragmentation, because fragmenting allocation patterns show a stable live-bytes count while RSS doubles. If your profiler says “live heap 340 MB, flat for 8 hours” and RSS is 2.4 GB, stop looking for a leak and start looking at the allocator.

The measurement that separates them is the ratio of RSS to mallinfo2().uordblks. Under 1.3 is normal. Between 1.5 and 2.0 is fragmentation worth tuning. Above 2.5, you have either a real leak outside the profiler’s view (mmap’d regions, thread stacks, JIT code) or a runaway arena count. Check /proc/$PID/maps for anonymous regions over 64 MB — those are usually arenas or mmap’d allocations bypassing the heap.

malloc_trim and the trim thresholds

malloc_trim(0) walks every arena and releases free pages above the top allocation that qualify. Called every minute from a timer thread, it holds RSS to a flat line in most services, at the cost of a brief lock per arena. After each trim, the next allocation re-faults pages from the kernel, so calling it in a hot path costs you allocation latency you did not have before.

Alternatives: raise M_TRIM_THRESHOLD via mallopt so trimming happens on frees, at the cost of more brk calls; lower M_MMAP_THRESHOLD so large allocations bypass arenas; or switch to jemalloc with MALLOC_CONF=dirty_decay_ms:1000,muzzy_decay_ms:0 so pages return within a second. Decay is the cleanest fix but worth benchmarking.

The container trap: OOM with “healthy” RSS

Signal	Meaning	Action
RSS/uordblks < 1.3	normal allocator overhead	none
1.5–2.0	arena fragmentation	cap arena count, timed trim
> 2.5	leak or runaway arenas	inspect maps, switch allocator
RSS near limit, uordblks low	OOM imminent, no leak	trim before the cgroup kills
RSS resets after restart	confirming fragmentation	trim schedule, arena cap

In cgroup v2, memory.current includes page cache charged to the container, and memory.max triggers reclaim (file pages first) before OOM. That gives an illusion of safety: the kernel drops cache, RSS keeps growing, and the container dies with oom_kill 1 in memory.events while the service reported no leak and heap profilers showed a flat live set. Add memory.high at 80% of memory.max to get throttling before the kill; it will slow the service rather than terminate it.

Fix ordering matters: cap arenas, then schedule trims, then consider a different allocator. Swapping allocators to fix unmeasured fragmentation is how a service gets 40 GB of resident overhead and a new class of tail latency. Change one thing, re-measure over 12 hours.

A 12-hour validation loop

Log RSS, uordblks, arena, and memory.current every 60 seconds. After any change, confirm RSS/uordblks stays under 1.4 through a full traffic cycle including the nightly batch. Drift during batch only means size-class-specific fragmentation in the batch code.

Fragmentation is a property of allocation patterns over time, not a bug to be patched out. Services that run for months need either an allocator that returns pages on a schedule or a trim routine that does it for them. Both are cheap; the OOM kill at 3 a.m. is not. If you are choosing an instance size, size the memory limit for your steady-state arena total plus 40% headroom rather than for the live heap, and this whole class of incident stops appearing on your dashboards.