{"id":1570,"date":"2026-11-02T02:10:51","date_gmt":"2026-11-02T02:10:51","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1570"},"modified":"2026-11-02T02:10:51","modified_gmt":"2026-11-02T02:10:51","slug":"malloc-warehouse-fragmentation-rss-growth","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/","title":{"rendered":"Why a Process&#8217;s RSS Grows for Hours Without a Leak"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind reports nothing, <code>gdb<\/code> finds no growing references, and <code>heaptrack<\/code> shows total allocations flat. The process is not leaking. It is fragmenting, and the allocator is hoarding pages that it has already freed from your point of view.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This pattern accounts for many &#8220;mystery OOM&#8221; incidents on long-lived services in memory-limited cgroups. glibc&#8217;s malloc does not give memory back eagerly; it keeps freed chunks ready for reuse, because reuse is cheaper than a syscall. The trade is RSS that only rises until something forces a trim \u2014 often the OOM killer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The fixes have side effects, so understanding the mechanism matters. A new arena per thread can multiply RSS by 8\u00d7 on a 64-thread process; dropping <code>MALLOC_ARENA_MAX<\/code> to 1 can serialise a previously lock-free workload. <a href=\"https:\/\/virtualserversvps.com\/\">When the memory cgroup limit is the real ceiling<\/a>, both mistakes are fatal in opposite directions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How glibc malloc actually organises memory<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Below 128 KB (the <code>M_MMAP_THRESHOLD<\/code> default), an allocation comes from an arena: a 64 MB reserved region grown with <code>brk<\/code> for the main arena and <code>mmap<\/code> for thread arenas. Inside an arena, free chunks of different sizes coalesce only with adjacent neighbours; a 40-byte chunk wedged between two live 1 MB chunks fragments that 64 MB region for the rest of the process&#8217;s life. The arena cannot reclaim the middle.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Then there is the second layer: only the top chunk of an arena can be returned with <code>brk<\/code>, and only if it exceeds <code>M_TRIM_THRESHOLD<\/code> (default 128 KB). A process that allocates and frees in a pattern that leaves a live chunk near the top will never satisfy that condition. This is why RSS grows in steps and never comes back down: the arena grew, the top chunk is pinned, and every subsequent allocation reuses the holes first.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># see arenas and fragmentation at runtime\nMALLOC_ARENA_MAX=4 MALLOC_STATS=1 .\/svc\nmallinfo2()  # uordblks vs fordblks vs hblkhd\ncat \/proc\/$PID\/smaps_rollup | egrep 'Rss|Pss'\n\n# force a trim without restarting\npython3 -c \"import ctypes;ctypes.CDLL('libc.so.6').malloc_trim(0)\"<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Why threads multiply the problem<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Each thread that allocates and finds the main arena contended gets its own arena, up to 8 \u00d7 <code>nproc<\/code> on 64-bit. A 32-thread service on a 16-core host can end up with 32 arenas, each of which may grow to several hundred MB under bursty allocation. <code>mallinfo2().arena<\/code> divided by 1024 and summed per arena shows the total; if that number tracks RSS closely, fragmentation is the driver.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Setting <code>MALLOC_ARENA_MAX=4<\/code> caps the count and is usually the highest-value single change. The cost is lock contention: measure throughput before and after. Most request-serving services lose under 3%; a parser doing millions of small allocations per second can lose 20%, in which case jemalloc with per-thread caches is the better answer.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">free() returns to the arena, not to the kernel<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When <code>free()<\/code> is called, the chunk goes back to the arena&#8217;s free list. The kernel is not involved. The page backing that chunk is not <code>madvise<\/code>d away unless <code>MADV_DONTNEED<\/code> trimming triggers, which happens only at the top of the arena or during an explicit <code>malloc_trim<\/code>. So <code>free()<\/code> of a 900 MB buffer of 64 KB chunks can leave RSS unchanged, and <code>top<\/code> will report the process as still holding that memory.<\/p>\n\n\n<p class=\"wp-block-paragraph\">This is correct behaviour for a general-purpose allocator optimising for speed, and wrong for a container that is billed by RSS. It also explains why RSS is a lagging indicator: tools like <code>ps<\/code> show pages the allocator owns, not bytes your code can reach. The gap between those two numbers is the fragmentation you need to measure, and <code>uordblks<\/code> versus <code>arena<\/code> from <code>mallinfo2<\/code> gives it to you directly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">When to trust heap profiling, and when not to<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Heap profilers attribute live allocations to call sites. They are excellent at finding a genuinely retained object graph and useless at finding fragmentation, because fragmenting allocation patterns show a stable live-bytes count while RSS doubles. If your profiler says &#8220;live heap 340 MB, flat for 8 hours&#8221; and RSS is 2.4 GB, stop looking for a leak and start looking at the allocator.<\/p>\n\n\n<p class=\"wp-block-paragraph\">The measurement that separates them is the ratio of RSS to <code>mallinfo2().uordblks<\/code>. Under 1.3 is normal. Between 1.5 and 2.0 is fragmentation worth tuning. Above 2.5, you have either a real leak outside the profiler&#8217;s view (mmap&#8217;d regions, thread stacks, JIT code) or a runaway arena count. Check <code>\/proc\/$PID\/maps<\/code> for anonymous regions over 64 MB \u2014 those are usually arenas or mmap&#8217;d allocations bypassing the heap.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">malloc_trim and the trim thresholds<\/h2>\n\n\n<p class=\"wp-block-paragraph\"><code>malloc_trim(0)<\/code> walks every arena and releases free pages above the top allocation that qualify. Called every minute from a timer thread, it holds RSS to a flat line in most services, at the cost of a brief lock per arena. After each trim, the next allocation re-faults pages from the kernel, so calling it in a hot path costs you allocation latency you did not have before.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Alternatives: raise <code>M_TRIM_THRESHOLD<\/code> via <code>mallopt<\/code> so trimming happens on frees, at the cost of more <code>brk<\/code> calls; lower <code>M_MMAP_THRESHOLD<\/code> so large allocations bypass arenas; or switch to jemalloc with <code>MALLOC_CONF=dirty_decay_ms:1000,muzzy_decay_ms:0<\/code> so pages return within a second. Decay is the cleanest fix but worth benchmarking.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The container trap: OOM with &#8220;healthy&#8221; RSS<\/h2>\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Signal<\/th><th>Meaning<\/th><th>Action<\/th><\/tr><\/thead><tbody><tr><td>RSS\/uordblks &lt; 1.3<\/td><td>normal allocator overhead<\/td><td>none<\/td><\/tr><tr><td>1.5\u20132.0<\/td><td>arena fragmentation<\/td><td>cap arena count, timed trim<\/td><\/tr><tr><td>&gt; 2.5<\/td><td>leak or runaway arenas<\/td><td>inspect maps, switch allocator<\/td><\/tr><tr><td>RSS near limit, uordblks low<\/td><td>OOM imminent, no leak<\/td><td>trim before the cgroup kills<\/td><\/tr><tr><td>RSS resets after restart<\/td><td>confirming fragmentation<\/td><td>trim schedule, arena cap<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p class=\"wp-block-paragraph\">In cgroup v2, <code>memory.current<\/code> includes page cache charged to the container, and <code>memory.max<\/code> triggers reclaim (file pages first) before OOM. That gives an illusion of safety: the kernel drops cache, RSS keeps growing, and the container dies with <code>oom_kill 1<\/code> in <code>memory.events<\/code> while the service reported no leak and heap profilers showed a flat live set. Add <code>memory.high<\/code> at 80% of <code>memory.max<\/code> to get throttling before the kill; it will slow the service rather than terminate it.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Fix ordering matters: cap arenas, then schedule trims, then consider a different allocator. Swapping allocators to fix unmeasured fragmentation is how a service gets 40 GB of resident overhead and a new class of tail latency. Change one thing, re-measure over 12 hours.<\/p>\n\n\n<h2 class=\"wp-block-heading\">A 12-hour validation loop<\/h2>\n\n\n<p class=\"wp-block-paragraph\">Log <code>RSS<\/code>, <code>uordblks<\/code>, <code>arena<\/code>, and <code>memory.current<\/code> every 60 seconds. After any change, confirm RSS\/uordblks stays under 1.4 through a full traffic cycle including the nightly batch. Drift during batch only means size-class-specific fragmentation in the batch code.<\/p>\n\n\n<p class=\"wp-block-paragraph\">Fragmentation is a property of allocation patterns over time, not a bug to be patched out. Services that run for months need either an allocator that returns pages on a schedule or a trim routine that does it for them. Both are cheap; the OOM kill at 3 a.m. is not. If you are choosing an instance size, <a href=\"https:\/\/virtualserversvps.com\/\">size the memory limit for your steady-state arena total plus 40% headroom<\/a> rather than for the live heap, and this whole class of incident stops appearing on your dashboards.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":1,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-1570","post","type-post","status-publish","format-standard","hentry","category-provider-reviews-comparisons"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Why a Process&#039;s RSS Grows for Hours Without a Leak - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind reports\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why a Process&#039;s RSS Grows for Hours Without a Leak\" \/>\n<meta property=\"og:description\" content=\"Why a Process&#039;s RSS Grows for Hours Without a Leak\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-02T02:10:51+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/\",\"name\":\"Why a Process's RSS Grows for Hours Without a Leak - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-11-02T02:10:51+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind reports\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Why a Process&#8217;s RSS Grows for Hours Without a Leak\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Why a Process's RSS Grows for Hours Without a Leak - Virtual Servers VPS Blog","description":"A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind reports","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/","og_locale":"en_US","og_type":"article","og_title":"Why a Process's RSS Grows for Hours Without a Leak","og_description":"Why a Process's RSS Grows for Hours Without a Leak","og_url":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-11-02T02:10:51+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/","url":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/","name":"Why a Process's RSS Grows for Hours Without a Leak - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-11-02T02:10:51+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"A service that has been up for three days is at 1.2 GB RSS. Six hours later it is at 2.7 GB, then it plateaus, then it climbs again. Valgrind reports","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/malloc-warehouse-fragmentation-rss-growth\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Why a Process&#8217;s RSS Grows for Hours Without a Leak"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1570","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1570"}],"version-history":[{"count":1,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1570\/revisions"}],"predecessor-version":[{"id":1571,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1570\/revisions\/1571"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1570"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1570"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1570"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}