jemalloc Arena and Decay Tuning for Redis and nginx on Memory-Constrained Instances

Long-running daemons on small instances accumulate memory that the allocator never returns to the kernel. The process RSS climbs for days while the application’s own live heap stays flat, and eventually the OOM killer picks off whichever process has the highest oom_score. The usual cause is glibc’s default arena behaviour: as worker threads allocate, each acquires its own arena, and freed memory fragments into pages that are no longer contiguous enough to return via malloc_trim. Linking jemalloc and tuning its decay parameters is the standard fix, and it is measurable in both RSS and p99 latency.

This walkthrough uses Redis and nginx as the two reference cases because both are multithreaded and both are commonly co-located on the same premium cloud solutions instance. The method applies to any C or C++ service you build yourself.

Confirm that the allocator is actually the problem

Start with RSS trend rather than a single snapshot. Pull VmRSS every minute for an hour alongside the application’s internal heap metric (Redis INFO memory → used_memory, or your runtime’s equivalent):

while true; do ts=$(date +%s); rss=$(grep VmRSS /proc/$(pgrep -o redis-server)/status | awk '{print $2}'); used=$(redis-cli info memory | awk -F: '/^used_memory:/{print $2}' | tr -d '\r'); echo "$ts $rss $used"; sleep 60; done

If RSS grows by tens of megabytes per day while used_memory is flat within a few percent, allocator retention is the explanation. Confirm with malloc_stats() semantics by inspecting MALLOC_ARENA_MAX behaviour — if there are many arenas and a large RSS-to-live ratio, stop looking elsewhere.

Signal	Allocator retention	Real leak
RSS trend	Steps up during load, partially returns	Monotonic, never returns
Live heap metric	Flat	Rising
After load stops	Plateau, sometimes falls	Keeps climbing
Fragmentation ratio	mem_fragmentation_ratio > 1.5	Ratio near 1.0 but used_memory rising

Install and link jemalloc

On Debian and Ubuntu the package is libjemalloc2 and the shared object lands at /usr/lib/x86_64-linux-gnu/libjemalloc.so.2. Verify before touching anything:

apt install -y libjemalloc2
ls -l /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
ldd $(which redis-server) | grep -i jemalloc

Redis ships with jemalloc linked by default in official builds; if ldd shows it, you are already using the right allocator and the problem is configuration, not linking. For a custom binary or a C service of your own, preload the library instead of rebuilding:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 /usr/local/bin/myapp

For a systemd unit, set it in the unit rather than in a shell wrapper so it survives restarts:

[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"
Environment="MALLOC_CONF=dirty_decay_ms:5000,muzzy_decay_ms:1000,background_thread:true"

Run systemctl daemon-reload && systemctl restart redis-server and check cat /proc/$(pgrep -o redis-server)/maps | grep jemalloc. If the library is not mapped, the preload failed silently — systemd strips LD_* variables for setuid binaries, so this only works for ordinary services.

Understand the decay parameters before choosing values

jemalloc returns memory in two stages: dirty pages are freed to the OS periodically, while muzzy pages are reclaimed lazily. Long decay windows improve throughput by reusing hot pages, but let RSS stay high after bursts. Short windows lower RSS but force more madvise calls.

• dirty_decay_ms:5000 — returns freed pages within a few seconds; suitable for a 4 GB host with a hard memory budget.
• dirty_decay_ms:1000 — aggressive; use on memory-constrained instances where RSS spikes trigger OOM.
• dirty_decay_ms:30000 — default; better for steady-state throughput on hosts with headroom.
• background_thread:true — moves purging to worker threads; reduces latency spikes caused by synchronous malloc_trim.
• narenas:2 or narenas:4 — on a 2–4 vCPU host, more arenas than cores adds fragmentation without any parallel benefit.

Validate a config string before deploying it: MALLOC_CONF="dirty_decay_ms:5000" /bin/true; echo $? returns 1 on a syntax error. Silent typos are common, and jemalloc falls back to defaults without complaint in some builds.

Measure the effect with a controlled burst

An allocator change is only justified if it reduces RSS after a burst without hurting p99 latency. Run the same load before and after:

1. Record baseline RSS and redis-cli --latency-history -i 5 for 5 minutes idle.
2. Push a write-heavy burst: redis-benchmark -n 2000000 -c 50 -t set,get,incr -r 1000000 -q.
3. Stop the burst and wait 60 seconds.
4. Compare RSS 60 seconds after the burst with RSS before it.
5. Compare latency-history max values; a regression there means the decay windows are too aggressive for your workload.

On this test shape, glibc typically retains 15–30% of burst memory after the load stops; jemalloc with a 5-second dirty decay returns most of it within a minute, at the cost of roughly 2–4% throughput during the burst itself. For a process whose survival depends on staying under a cgroup memory cap, that trade is almost always correct.

Apply the same reasoning to nginx workers

nginx workers are single-threaded, so arena proliferation is less of an issue, but large proxy_buffers and persistent SSL sessions still fragment. More practically, if a worker’s RSS exceeds about 80 MB with modest traffic, look at buffer sizing before reaching for an allocator swap. Confirm with pmap -x $(pgrep -o nginx) | tail -1 and strace -c -p <pid> to see the syscall profile.

When the host is shared between several services, allocator tuning alone will not prevent one process from evicting the page cache another depends on. Enforce ceilings at the cgroup level as well, as described in the managed and unmanaged server operations comparison. Allocators decide how fast a process gives memory back; cgroups decide what happens when it does not.

A checklist for production rollout

• Set MALLOC_CONF in the unit file, not in a shell profile.
• Verify the library is mapped in /proc/<pid>/maps after restart.
• Keep narenas at or below the vCPU count.
• Log RSS hourly to a file so the next regression is visible in a diff rather than at 3 a.m.
• Re-run the burst test after any upstream version bump; default decay values change between jemalloc releases.

The end state is a process whose RSS tracks its live data set instead of its historical peak. That is the property you actually want, and it is measurable with three commands and one load test.