Keeping VPS Logs Under Control: A Step-by-Step journald and logrotate Setup

A VPS disk filling up with logs is one of the most common causes of sudden downtime. The symptom looks dramatic — services fail to write, databases refuse transactions — but the cause is mundane: journald is retaining everything forever, or logrotate is configured but never actually trimming a chatty application’s log. This is a concrete walkthrough for sizing, rotating, and verifying both systems so your disk stays predictable.

You need a host with enough headroom to survive a logging incident in the first place. If your current plan is tight on disk, review the storage characteristics of the VPS hosting plans on our main site before tuning retention windows.

Prerequisites

• Debian/Ubuntu or RHEL-family VPS with systemd and journald
• logrotate installed (usually present by default)
• Root or sudo access
• Knowledge of your total disk size and current free space

Step 1: Measure Before You Configure

Never set a retention window without knowing what logs currently consume.

df -h /
journalctl --disk-usage
du -sh /var/log /var/log/journal 2>/dev/null
du -sh /var/log/* 2>/dev/null | sort -h | tail -10

Note the journal size and the top consumers. A well-behaved small VPS keeps the journal under a few hundred megabytes; if it is multiple gigabytes, retention is the first thing to fix.

Step 2: Cap journald by Size and Time

Edit the journald configuration. Both size and age limits are enforced, and whichever is hit first wins — which is what you want.

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/99-limits.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
SystemMaxFileSize=50M
MaxRetentionSec=1month
Compress=yes
RateLimitIntervalSec=30s
RateLimitBurst=1000
EOF

sudo systemctl restart systemd-journald
journalctl --disk-usage

# Immediately reclaim if the journal is already oversized
sudo journalctl --vacuum-size=400M
sudo journalctl --vacuum-time=2weeks

SystemKeepFree=1G is the safety valve: journald will stop growing before it consumes the last gigabyte of the filesystem, keeping the host alive even under a log flood. RateLimitBurst prevents a single crashing service from spamming the journal thousands of times per second.

Step 3: Configure Application Log Rotation

journald does not manage plain files under /var/log. Those are logrotate’s job. Create a dedicated policy rather than editing the vendor defaults.

sudo tee /etc/logrotate.d/myapp > /dev/null <<'EOF'
/var/log/myapp/*.log {
    daily
    rotate 14
    size 50M
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
    dateext
    dateformat -%Y%m%d
    create 0640 www-data adm
    sharedscripts
    postrotate
        systemctl reload myapp.service > /dev/null 2>&1 || true
    endscript
}
EOF

sudo logrotate -d /etc/logrotate.d/myapp   # dry run, shows decisions
sudo logrotate -f /etc/logrotate.d/myapp   # force once to confirm

Two details matter. copytruncate lets you rotate logs from programs that cannot be told to reopen their file (it copies then truncates in place), at the cost of a tiny window where a few lines can be lost. If your service supports a reload signal, drop copytruncate and rely on the postrotate reload for zero-loss rotation. size 50M combined with daily means a log escalates to rotation on either trigger.

Step 4: Confirm the Timer Is Actually Running

A common failure is a perfect logrotate config that nothing ever invokes.

systemctl status logrotate.timer
systemctl list-timers | grep logrotate
cat /etc/logrotate.conf | grep -v '^#' | grep -v '^$'
cat /var/lib/logrotate/status | head -20

The status file records the last rotation date per log. If a log you expect is missing from it, logrotate is not matching the path — check your globs against the real filenames.

Step 5: Optional Centralization

Once local retention is sane, forwarding is cheap and preserves history after a host dies. A minimal rsyslog forward:

sudo tee /etc/rsyslog.d/60-forward.conf > /dev/null <<'EOF'
*.* action(type="omfwd" target="logs.internal" port="514" protocol="tcp"
           action.resumeRetryCount="-1" queue.type="linkedList"
           queue.size="10000")
EOF
sudo systemctl restart rsyslog

The linkedList disk-assisted queue means local logging survives a network outage to the collector instead of blocking. Keep local retention even with forwarding — the collector is an archive, not a substitute.

Verification Step

Prove the limits are enforced rather than merely configured.

# 1. Journal honors the size cap
sudo dd if=/dev/zero bs=1M count=800 | xargs -I{} sh -c   'logger -p user.info "flood test payload"'
sleep 5
journalctl --disk-usage      # must remain near SystemMaxUse, not climb past it

# 2. logrotate policy parses and rotates
sudo logrotate -d /etc/logrotate.d/myapp 2>&1 | grep -E 'rotating|skipping'
ls -lh /var/log/myapp/

# 3. Free space did not collapse
df -h /

If journalctl --disk-usage stays pinned near your cap after the flood, the limit is working. Rotation is confirmed when you see a dated archive alongside the fresh file.

Troubleshooting

Symptom	Cause	Fix
Journal grows past SystemMaxUse	Config not in the drop-in dir	Verify with systemd-analyze cat-config systemd/journald.conf
Rotation runs but files never shrink	Logs outside the configured glob	Match exact paths; check /var/lib/logrotate/status
logrotate -d reports no matching logs	Wrong directory or filename pattern	Test the glob with ls first
Lines lost at rotation	copytruncate race	Remove it and reload the service in postrotate
Timer never fires	logrotate.timer disabled	systemctl enable --now logrotate.timer
Journal restarts lose entries	Storage=volatile	Set Storage=persistent and create /var/log/journal

With size caps on the journal, an explicit rotation policy for application logs, and a verification pass after the flood test, log growth becomes a bounded, predictable cost instead of a surprise outage.