inotify Watch Limits on a Small VPS: Finding the Leaky Service Before It Breaks Backups

There is a specific way a small VPS stops taking backups: not with an alarm, but with a quiet ENOSPC from inotify_add_watch in a service that has been running for 40 days. The kernel watch table is a shared, finite resource — fs.inotify.max_user_watches defaults to 8,192 on many distros — and every file watcher on the box draws from the same pool. This guide shows how to attribute consumption per process, find the leak, and fix it instead of raising the limit until the next outage.

The failure is silent at its origin and noisy downstream. A sync daemon that cannot add a watch does not crash; it logs once and stops noticing changes. Your backup tool then copies a stale tree, exits zero, and monitoring stays green. The discipline is the same as auditing SUID binaries and file capabilities: measure real state before changing sysctl values.

What the Watch Table Actually Is

inotify is a per-instance object model in the kernel. An instance is the file descriptor from inotify_init1(). Inside each instance you register watches, each bound to one inode and a mask of events. Two independent limits govern this: instances and watches, both capped per UID — not per process. Every service running as root therefore shares a single budget.

That is the whole trap on a small VPS. Docker, systemd units, a log shipper, a config agent and your backup hook usually all run as root. One misbehaving directory walker registering a watch per file can consume the entire root budget and starve a service that wanted 200 watches. Memory is real too: each watch pins kernel structures of roughly 1 KB with its inode reference, so a million watches is on the order of a gigabyte of unswappable kernel memory.

Reading the Limits and Current Usage

Start with the three sysctls that set the ceiling: fs.inotify.max_user_watches, fs.inotify.max_user_instances and fs.inotify.max_queued_events. There is no single per-UID counter in /proc, but you can derive consumption exactly by summing watch counts from each process’s fdinfo and cross-checking total slab memory.

The fdinfo approach is the most reliable because it attributes watches to a PID and therefore to a unit. For each inotify descriptor a process holds, the kernel lists every watch as an inotify wd: line. Counting those lines per PID gives a ranked table of who is consuming the shared budget right now.

Counting Watches Per Process

The loop below walks every PID, finds descriptors whose fdinfo contains inotify entries, and totals them. Run it as root, since /proc/<pid>/fdinfo is only readable by the owning user. Output is sorted, so the leak is at the top.

# Current kernel limits
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances fs.inotify.max_queued_events

# Kernel memory currently charged to inotify objects
grep -i inotify /proc/slabinfo

# Watches and instances per PID (run as root)
for p in /proc/[0-9]*; do
  pid=${p#/proc/}
  w=0; i=0
  for f in "$p"/fdinfo/*; do
    [ -r "$f" ] || continue
    if grep -q '^inotify' "$f" 2>/dev/null; then
      i=$((i+1))
      w=$((w + $(grep -c '^inotify wd:' "$f")))
    fi
  done
  [ "$w" -gt 0 ] && printf '%8d %6d %s\n' "$w" "$i" "$(cat "$p/comm" 2>/dev/null)"
done | sort -rn | head -20

# Map the top PIDs back to systemd units
systemctl status $(ps -o unit= -p <PID>) --no-pager | head -5

# Watch a tree live and see the event rate it generates
inotifywait -m -r -e create,delete,modify --format '%w%f %e' /var/www

Two things to watch. Instance count versus watch count: a process holding 60 instances with 3 watches each is an fd leak; one holding 1 instance with 300,000 watches is a recursive walker. And the slabinfo line — if inotify slab memory grows steadily, you have a leak, and raising the ceiling only delays the failure.

The Failure Modes You Will Actually See

The most common leak is a recursive watcher pointed at a directory that accumulates files: a mail spool, a Docker overlay root, a log directory, or a web root with session files. Each new file adds a watch; deletions may not remove it if the watcher mishandles IN_IGNORED. After weeks the process holds tens of thousands of watches for files deleted months ago.

The second is an instance leak: a daemon calling inotify_init() on every reconnect and never closing the descriptor. That exhausts max_user_instances at 128 long before watches run out, and surfaces as Too many open files from an unrelated service sharing the UID. Both are visible in the per-PID table, which is why attribution beats the raw total.

Why Backup Failures Are the Canary

Backup tools are the classic victim because they are last in the chain. Restic, borg, or a monitoring agent using change detection needs dozens to hundreds of watches. If the root budget is 99% consumed by a long-running watcher, the backup process gets ENOSPC on its first inotify_add_watch, logs a warning, and proceeds to take a full rather than incremental backup — or skips the watched path entirely.

Detection is a schedule correlation: backups succeed for weeks, then start running full every night or report zero changed files. Nobody notices until a restore is needed. The watch table belongs on the same dashboard as disk usage and inode counts — a quietly consumable resource with a hard ceiling.

Choosing a New Limit With Numbers

Do not set 524288 because a blog post said so. Size against measured demand: take the largest legitimate consumer, multiply by three for growth, round to a power-of-two-ish value. A 2 GB VPS serving a few sites rarely needs more than 131072; a container host may need 524288. Each watch costs unreclaimable kernel memory, so on a 1 GB instance an over-large limit is itself a stability risk.

Setting	Common default	Recommended small VPS	Cost of being wrong
max_user_watches	8192	131072	ENOSPC for backup tools
max_user_instances	128	1024	fd leak starves other units
max_queued_events	16384	32768	IN_Q_OVERFLOW, missed events
Watch slab memory	—	<2% of RAM	Unswappable memory pressure

Track the ratio of consumed to limit over time. Flat means correctly sized. A monotonic climb means a leak, and a sysctl bump buys days, not months — restart the offending unit on a timer, or fix the watcher to use one non-recursive watch plus periodic rescans.

A Minimal Guard Rail

A ten-line check in a systemd timer turns this from a surprise into a trend. Count total watches across all PIDs, compare to the sysctl value, alert above 80%, and include the top three consuming commands in the alert body so nobody has to SSH in to start investigating. Persist tuned values in /etc/sysctl.d/60-inotify.conf so a reboot does not reset them.

Pair that with a monthly review of long-running watcher processes and you catch the leak while it is still a number on a graph rather than a backup that silently stopped changing. For the broader ceiling picture on one box, measuring page-cache hit ratio with vmtouch and cachestat follows the same measure-then-tune method.