Disk Space Forensics on a Linux VPS: Where the Gigabytes Went

A full disk on a production instance does not announce itself politely. Services fail to write logs, the database refuses new connections with “no space left on device”, and PHP returns a 500 with nothing in the error log because the error log itself could not be written. The first symptom is usually a monitor alert or a user complaint, minutes after the actual threshold was crossed.

Reclaiming space is straightforward once you know where it went. The hard part is that the usual suspect — big files in the usual places — is often wrong. This procedure attributes usage systematically, then caps the two things that regrow on their own so the problem does not return next month.

Confirm It Is a Space Problem, Not an Inode Problem

Before deleting anything, check whether you have run out of bytes or out of inodes. A directory with ten million tiny session files reports 100% inode use while showing plenty of free space in df -h. The fix for that is completely different.

df -h /
df -i /
stat -f /

If df -h shows space free but writes fail, the culprit is often a deleted file still held open by a process. lsof +L1 lists exactly those; restarting the holder releases the blocks without any file system cleanup. This case is surprisingly common with log rotation misconfigurations, where the writer keeps its descriptor on the rotated-and-deleted original.

Attribute Usage by Directory, Sorted

Skip du -sh /* on a large disk, which walks everything and can take minutes. Instead, do one level at a time, sorted, excluding other mounts so a mounted backup volume does not dominate the output.

sudo du -x -h -d1 / 2>/dev/null | sort -rh | head -15
sudo du -x -h -d1 /var 2>/dev/null | sort -rh | head -15
# then drill into the top directory, e.g.:
sudo du -x -h -d1 /var/lib 2>/dev/null | sort -rh | head -15

On a typical web instance the top three answers are /var/lib/mysql, /var/log and the application’s upload or cache directory. If a container runtime is in use, /var/lib/docker frequently outranks all three, and the layers there are not reachable by ordinary du intuition — that deserves its own approach.

Cap the Journal So It Cannot Regrow

systemd-journald defaults allow the journal to consume up to 10% of the filesystem, which on a 160 GB disk is 16 GB of logs nobody reads. Vacuum it now and set a hard ceiling so it stops growing back.

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=2week

Then run systemctl restart systemd-journald. Note that the journal and the traditional /var/log files are separate consumers: logrotate handles syslog, nginx access logs and application logs, and it needs its own retention settings. Rotation without compression leaves months of uncompressed history behind.

Find the Million-File Case

Some bloat is invisible to du-by-size because it is made of enormous numbers of small files: session stores, cache fragments, mail queues, thumbnail variants. When inode usage is high but no single file is large, count files per directory instead.

sudo find /var -xdev -type d -printf '%p\n' \
  | sudo xargs -I{} sh -c 'echo "$(find "{}" -maxdepth 1 | wc -l) {}"' \
  | sort -rn | head -15

Once identified, delete in bounded batches rather than one giant rm -rf, which can spike I/O and leave the directory half-emptied if interrupted. find dir -type f -mtime +30 -delete with a time filter is safer because it removes only what has aged out, and it can be run on a schedule.

Clear Container and Package Layer Bloat

Docker keeps every image layer it has ever pulled or built. A handful of repeated rebuilds of a 2 GB image can consume more space than the database. The same is true of package manager caches after a large upgrade.

docker system df
docker image prune -a        # remove unused images, not volumes
docker builder prune
sudo journalctl --vacuum-size=500M
sudo apt clean               # or: dnf clean all

Never run docker system prune --volumes casually on a host with stateful containers; it deletes anonymous volumes, which is where database containers commonly store their data. The reclaim from images and builds alone is usually enough to buy back tens of gigabytes.

Find Stale Large Files and Old Backups

Backup archives that were meant to be uploaded and deleted, old tarballs from a long-finished migration, and rotated database dumps accumulate silently. Sort by size across the whole filesystem and look at the top twenty.

sudo find / -xdev -type f -size +200M -printf '%s %TY-%Tm-%Td %p\n' \
  | sort -rn | head -20

Check the modification dates against the names before deleting. If the found files are backups, the real problem is that the backup script has no retention policy — deleting the files once means repeating this exercise next quarter. Retention design is a separate topic, and provisioning headroom for it is one of the practical arguments in the case for cloud VPS storage that scales independently.

Verify the Reclaim and Add a Guard Rail

After cleanup, re-check df -h and df -i, then add a monitor that warns at 80% rather than 95%. A disk alert at 95% leaves no time to act during business hours, and the failure mode of a full disk is rarely graceful.

A simple cron job writing usage to the journal is enough to trend the growth rate. If the disk climbs 2 GB per week, the fix is not more cleanup — it is either log retention, a database maintenance task, or a larger volume. When the choice is between cleaning weekly and moving to a bigger plan, the sizing trade-offs are covered in how VPS resources differ from shared hosting limits, where storage quotas are the first thing to feel artificial.