{"id":1509,"date":"2026-10-25T23:16:22","date_gmt":"2026-10-25T23:16:22","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1509"},"modified":"2026-10-25T23:16:22","modified_gmt":"2026-10-25T23:16:22","slug":"tripwire-audit-inotify-watch-limits-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/","title":{"rendered":"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet <code>ENOSPC<\/code> from <code>inotify_add_watch<\/code> in a service that has been running for 40 days. The kernel watch table is a shared, finite resource \u2014 <code>fs.inotify.max_user_watches<\/code> defaults to 8,192 on many distros \u2014 and every file watcher on the box draws from the same pool. This guide shows how to attribute consumption per process, find the leak, and fix it instead of raising the limit until the next outage.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The failure is silent at its origin and noisy downstream. A sync daemon that cannot add a watch does not crash; it logs once and stops noticing changes. Your backup tool then copies a stale tree, exits zero, and monitoring stays green. The discipline is the same as <a href=\"https:\/\/virtualserversvps.com\/blog\/auditing-suid-binaries-file-capabilities-vps\">auditing SUID binaries and file capabilities<\/a>: measure real state before changing sysctl values.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What the Watch Table Actually Is<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">inotify is a per-instance object model in the kernel. An instance is the file descriptor from <code>inotify_init1()<\/code>. Inside each instance you register watches, each bound to one inode and a mask of events. Two independent limits govern this: instances and watches, both capped per UID \u2014 not per process. Every service running as root therefore shares a single budget.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That is the whole trap on a small VPS. Docker, systemd units, a log shipper, a config agent and your backup hook usually all run as root. One misbehaving directory walker registering a watch per file can consume the entire root budget and starve a service that wanted 200 watches. Memory is real too: each watch pins kernel structures of roughly 1 KB with its inode reference, so a million watches is on the order of a gigabyte of unswappable kernel memory.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Reading the Limits and Current Usage<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Start with the three sysctls that set the ceiling: <code>fs.inotify.max_user_watches<\/code>, <code>fs.inotify.max_user_instances<\/code> and <code>fs.inotify.max_queued_events<\/code>. There is no single per-UID counter in <code>\/proc<\/code>, but you can derive consumption exactly by summing watch counts from each process&#8217;s <code>fdinfo<\/code> and cross-checking total slab memory.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The fdinfo approach is the most reliable because it attributes watches to a PID and therefore to a unit. For each inotify descriptor a process holds, the kernel lists every watch as an <code>inotify wd:<\/code> line. Counting those lines per PID gives a ranked table of who is consuming the shared budget right now.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Counting Watches Per Process<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The loop below walks every PID, finds descriptors whose fdinfo contains inotify entries, and totals them. Run it as root, since <code>\/proc\/&lt;pid&gt;\/fdinfo<\/code> is only readable by the owning user. Output is sorted, so the leak is at the top.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Current kernel limits\nsysctl fs.inotify.max_user_watches fs.inotify.max_user_instances fs.inotify.max_queued_events\n\n# Kernel memory currently charged to inotify objects\ngrep -i inotify \/proc\/slabinfo\n\n# Watches and instances per PID (run as root)\nfor p in \/proc\/[0-9]*; do\n  pid=${p#\/proc\/}\n  w=0; i=0\n  for f in \"$p\"\/fdinfo\/*; do\n    [ -r \"$f\" ] || continue\n    if grep -q '^inotify' \"$f\" 2&gt;\/dev\/null; then\n      i=$((i+1))\n      w=$((w + $(grep -c '^inotify wd:' \"$f\")))\n    fi\n  done\n  [ \"$w\" -gt 0 ] &amp;&amp; printf '%8d %6d %s\\n' \"$w\" \"$i\" \"$(cat \"$p\/comm\" 2&gt;\/dev\/null)\"\ndone | sort -rn | head -20\n\n# Map the top PIDs back to systemd units\nsystemctl status $(ps -o unit= -p &lt;PID&gt;) --no-pager | head -5\n\n# Watch a tree live and see the event rate it generates\ninotifywait -m -r -e create,delete,modify --format '%w%f %e' \/var\/www<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two things to watch. Instance count versus watch count: a process holding 60 instances with 3 watches each is an fd leak; one holding 1 instance with 300,000 watches is a recursive walker. And the <code>slabinfo<\/code> line \u2014 if inotify slab memory grows steadily, you have a leak, and raising the ceiling only delays the failure.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Failure Modes You Will Actually See<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The most common leak is a recursive watcher pointed at a directory that accumulates files: a mail spool, a Docker overlay root, a log directory, or a web root with session files. Each new file adds a watch; deletions may not remove it if the watcher mishandles <code>IN_IGNORED<\/code>. After weeks the process holds tens of thousands of watches for files deleted months ago.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The second is an instance leak: a daemon calling <code>inotify_init()<\/code> on every reconnect and never closing the descriptor. That exhausts <code>max_user_instances<\/code> at 128 long before watches run out, and surfaces as <code>Too many open files<\/code> from an unrelated service sharing the UID. Both are visible in the per-PID table, which is why attribution beats the raw total.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why Backup Failures Are the Canary<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Backup tools are the classic victim because they are last in the chain. Restic, borg, or a monitoring agent using change detection needs dozens to hundreds of watches. If the root budget is 99% consumed by a long-running watcher, the backup process gets <code>ENOSPC<\/code> on its first <code>inotify_add_watch<\/code>, logs a warning, and proceeds to take a full rather than incremental backup \u2014 or skips the watched path entirely.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Detection is a schedule correlation: backups succeed for weeks, then start running full every night or report zero changed files. Nobody notices until a restore is needed. The watch table belongs on the same dashboard as disk usage and inode counts \u2014 a quietly consumable resource with a hard ceiling.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Choosing a New Limit With Numbers<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Do not set 524288 because a blog post said so. Size against measured demand: take the largest legitimate consumer, multiply by three for growth, round to a power-of-two-ish value. A 2 GB VPS serving a few sites rarely needs more than 131072; a container host may need 524288. Each watch costs unreclaimable kernel memory, so on a 1 GB instance an over-large limit is itself a stability risk.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Setting<\/th><th>Common default<\/th><th>Recommended small VPS<\/th><th>Cost of being wrong<\/th><\/tr><\/thead><tbody><tr><td><code>max_user_watches<\/code><\/td><td>8192<\/td><td>131072<\/td><td>ENOSPC for backup tools<\/td><\/tr><tr><td><code>max_user_instances<\/code><\/td><td>128<\/td><td>1024<\/td><td>fd leak starves other units<\/td><\/tr><tr><td><code>max_queued_events<\/code><\/td><td>16384<\/td><td>32768<\/td><td>IN_Q_OVERFLOW, missed events<\/td><\/tr><tr><td>Watch slab memory<\/td><td>\u2014<\/td><td>&lt;2% of RAM<\/td><td>Unswappable memory pressure<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Track the ratio of consumed to limit over time. Flat means correctly sized. A monotonic climb means a leak, and a sysctl bump buys days, not months \u2014 restart the offending unit on a timer, or fix the watcher to use one non-recursive watch plus periodic rescans.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A Minimal Guard Rail<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A ten-line check in a systemd timer turns this from a surprise into a trend. Count total watches across all PIDs, compare to the sysctl value, alert above 80%, and include the top three consuming commands in the alert body so nobody has to SSH in to start investigating. Persist tuned values in <code>\/etc\/sysctl.d\/60-inotify.conf<\/code> so a reboot does not reset them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Pair that with a monthly review of long-running watcher processes and you catch the leak while it is still a number on a graph rather than a backup that silently stopped changing. For the broader ceiling picture on one box, <a href=\"https:\/\/virtualserversvps.com\/blog\/measuring-page-cache-hit-ratio-vmtouch-cachestat\">measuring page-cache hit ratio with vmtouch and cachestat<\/a> follows the same measure-then-tune method.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet ENOSPC from inotify_add_watch in a service that has been running for&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":2,"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1509","post","type-post","status-publish","format-standard","hentry","category-performance-optimization"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet ENOSPC from inotify_add_watch in a service that has been\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups\" \/>\n<meta property=\"og:description\" content=\"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-25T23:16:22+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/\",\"name\":\"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-10-25T23:16:22+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet ENOSPC from inotify_add_watch in a service that has been\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups - Virtual Servers VPS Blog","description":"There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet ENOSPC from inotify_add_watch in a service that has been","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/","og_locale":"en_US","og_type":"article","og_title":"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups","og_description":"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups","og_url":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-10-25T23:16:22+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/","url":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/","name":"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-10-25T23:16:22+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet ENOSPC from inotify_add_watch in a service that has been","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/tripwire-audit-inotify-watch-limits-vps\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1509","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1509"}],"version-history":[{"count":1,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1509\/revisions"}],"predecessor-version":[{"id":1512,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1509\/revisions\/1512"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1509"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1509"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1509"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}