Disk Full With Free Space: Fixing Inode Exhaustion

TL;DR / Summary: On ext4, a disk can have gigabytes free and still refuse to accept a single byte. The limit is inodes, the fixed-size records that track files. A directory with a million session files or a runaway cache can exhaust them long before the space runs out.

When df says there is space and writes fail

The error is unmistakable: “No space left on device”. Then you run df -h and see thirty percent free. You delete a log file, try again, and the same error returns.

The next command is the one people forget: df -i. It reports inode usage instead of block usage, and on a healthy server the two numbers usually track each other. On the server that will not accept writes, the inode column is almost always at a hundred percent.

This failure mode is common on small VPS instances because providers often format the disk with a generous inode ratio, which is fine for large media files and terrible for applications that create endless tiny files.

What an inode actually is

An inode is a metadata record: owner, permissions, timestamps, and pointers to the data blocks. Every file and directory consumes exactly one. The contents can be empty and the inode is still used.

The total count is decided when the filesystem is created. On ext4 it is fixed by the bytes-per-inode setting, typically one inode per sixteen kilobytes, so a forty gigabyte volume holds roughly two and a half million inodes no matter how you use it.

XFS allocates inodes dynamically, which makes exhaustion less likely but not impossible. It also means you cannot simply count inodes with df -i in the same way, so the diagnostic path differs slightly.

Finding the directory with a million files

Start broad and narrow down. du --inodes -x -d 1 /var prints inode counts per top-level directory without crossing into other filesystems. The offender is usually obvious within one or two passes.

If du is too slow, try find /var -xdev -type f | wc -l on the subdirectories one at a time, starting with the largest. It is crude, but it gives an answer in seconds on a small machine.

Remember to stay on one filesystem with -xdev. Without it you will wander into mounted volumes and count the wrong disk, which produces a number that makes no sense and wastes ten minutes.

The usual culprits

PHP session files are the classic offender. A site with no session garbage collection collects one small file per visitor, and a busy crawler population can produce hundreds of thousands in a week.

Mail queues come second. A deferred message that keeps retrying leaves a queue file in /var/spool/postfix, and a compromised form can leave tens of thousands of them.

Then there are the modern caches: Composer, npm, pip, thumbnail generators, and container layers. Anything that writes one file per request or per package will eventually produce a directory that no human wants to look at.

Cleaning with find and care

Delete by age first, not by name. find /var/lib/php/sessions -type f -mtime +1 -delete removes sessions older than a day and leaves active ones alone. That single command resolves a surprising share of cases.

For mail, list before deleting. postqueue -p | tail -n 1 tells you how many messages are waiting, and postsuper -d ALL deferred clears the deferred queue. Never flush a queue you have not inspected, because the messages may include a real customer’s password reset.

For package caches, use the tool’s own command rather than rm -rf. npm cache clean --force and composer clear-cache are safer than deleting directories by hand and less likely to break a lock file.

Preventing the next flood

Monitoring has to include inodes. Most uptime checks watch CPU, memory, and disk space, which is exactly why this failure arrives unannounced. Add a check on df -i at eighty percent and alert on it.

Configure session garbage collection properly. In PHP, set session.gc_maxlifetime and the probability values so cleanup actually runs, or point sessions at Redis and stop creating files at all.

Keep logs rotating on a size limit, not only on a schedule. A single error loop can write a million lines overnight, and logrotate by day will not save you if the file grows faster than the rotation.

If the volume is small and the workload is file-heavy, rebuild it with more inodes or move the cache directory to a dedicated disk. That is a planned outage instead of an outage that chooses its own moment.

A single cron job can do the watching. Read df -i once an hour, parse the use percentage, and send a message when it crosses eighty. Two lines of shell plus the mail relay you already have are enough, and the alert arrives while there is still room to delete something calmly.

Keep the check in the same script that watches disk space, so nobody has to remember that a second limit exists.

Capacity is two numbers, not one

Think of a VPS disk as having two capacities: bytes and entries. Most guides only discuss the first, so the second becomes a mystery at the worst possible time.

Use the VPS hosting guides to plan a volume that fits your workload, then put an inode check into the same dashboard that already watches disk space.

One extra column in a monitoring view is cheap. Discovering the limit during a traffic spike is not.