Auditing systemd Journal Disk Usage on a Small VPS: Measure, Cap and Verify

On a small VPS with a 20–40 GB disk, an unmanaged systemd journal is one of the most common causes of a surprise full filesystem. Unlike logrotate-driven text logs, the journal is a binary store with its own retention logic, and its size is not obvious from du on /var/log until you know where to look. This is a measurement-first audit: find out what the journal actually holds, how fast it grows, then cap it deliberately.

Measure Before You Cap Anything

journalctl reports its own disk usage directly, which is more reliable than guessing from directory sizes:

journalctl --disk-usage
du -sh /var/log/journal /run/log/journal 2>/dev/null
ls -lh /var/log/journal/*/ | head

Two locations matter. /run/log/journal is a tmpfs store that vanishes on reboot; /var/log/journal is persistent. If only the former exists, your logs are not surviving reboots at all — a separate problem worth fixing if you need post-mortem forensics.

Find Who Is Flooding the Journal

A journal that balloons from a few hundred MB to several GB in a day is usually one misbehaving unit writing in a tight loop. Rank units by volume:

journalctl --since "24 hours ago" --output=json \
  | jq -r '.SYSLOG_IDENTIFIER // ._COMM' | sort | uniq -c | sort -rn | head -20

If jq is not installed and you want a lighter view, use journalctl --since "1 hour ago" | awk '{print $5}' | sort | uniq -c | sort -rn. Either way the goal is to identify the top talker. Common culprits are a failing service crash-looping, an application logging every request at debug level, or a kernel warning repeated thousands of times per minute.

Cap Retention in journald.conf

journald offers three independent limits — size, time, and free-space floor. Set all three; they interact, and whichever triggers first wins.

sudo tee /etc/systemd/journald.conf.d/size.conf <<'EOF'
[Journal]
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=2week
MaxFileSec=1week
Compress=yes
Storage=persistent
EOF
sudo systemctl restart systemd-journald

Directive	Purpose	Suggested value
SystemMaxUse	Hard ceiling on journal size	500M–1G
SystemKeepFree	Always leave disk free	1G
MaxRetentionSec	Delete entries older than	2week
MaxFileSec	Rotate a journal file after	1week
Compress	Compress rotated files	yes

SystemKeepFree is the safety net: even if a log flood occurs, journald stops before the disk fills and takes your services down with it. On a 20 GB VPS, keeping a 1 GB floor is cheap insurance.

Force Rotation and Reclaim Immediately

Config changes apply to new writes; existing data must be rotated and vacuumed explicitly. Do this in the right order:

sudo journalctl --rotate
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=2weeks
journalctl --disk-usage

--rotate seals the current active file; --vacuum-size then deletes oldest entries until the cap is met. Run --disk-usage afterward to confirm the reclaim actually happened.

Set Caps Per-Unit, Not Just Globally

One chatty unit can still consume the entire global budget and evict everything else. Limit the noisy service at the source with LogRateLimit*:

• Add LogRateLimitIntervalSec=30s and LogRateLimitBurst=1000 to the misbehaving unit.
• Set MaxLevelStore=warning on units that log debug noise you rarely need.
• If the volume is genuinely useful, send it to a remote collector instead of keeping it locally.

Automate Verification

Retention limits are only trustworthy if they hold over time. A weekly check that surfaces journal size alongside overall disk usage catches drift early:

journalctl --disk-usage
df -h / | awk 'NR==2 {print "root used:", $5}'

Wire that into an existing monitoring agent and alert when the journal exceeds, say, 600 MB against a 500 MB cap — a small overshoot is normal between vacuums, but a growing gap means a limit is not being applied. If disk pressure is a recurring theme on your instance, it is worth reviewing the comparison of VPS plans to see whether a larger volume would cost less than the operational effort of constant cleanup.

Keep the Useful Part, Discard the Rest

Capping the journal is not the same as losing observability. Most of what fills a journal is routine and can be sent elsewhere or suppressed, while the entries you actually need for debugging an incident are a small fraction. Before capping, decide what is worth keeping: kernel errors, service start and stop events, authentication failures, and OOM messages. What is rarely worth keeping at full fidelity is per-request application logging, which belongs in files managed by logrotate and shipped to an aggregator if it matters.

A useful pattern on a small VPS is to keep local retention short and forward anything important off-box. That way the instance’s disk stays predictable regardless of what a noisy service does, and you retain long-term history without paying for it in local storage. The journal becomes a rotating buffer for recent events; the remote store becomes the archive.

Audit Summary

1. Read journalctl --disk-usage for the true size.
2. Identify the top logging unit.
3. Set size, free-space, and time caps together.
4. Rotate and vacuum to apply them now.
5. Rate-limit noisy units at the source.
6. Monitor size weekly against the configured cap.

A journal is a diagnostic asset, not a data hoard. Capping it to a size you can afford, and knowing where the flood comes from, means a full disk is never a mystery. Combined with sane log retention practices across the rest of the system, this keeps a small VPS predictable under sustained write load.