Diagnosing Sustained Disk-Write Amplification on a VPS (and When to Add Monitoring)

A VPS that does nothing but still lights up its disk I/O counter is a common and frustrating problem. The writes are real, but the process responsible usually is not the one you would guess. This article walks through a systematic method to trace sustained write amplification to its source, and discusses where a self-hosted monitoring and uptime stack helps versus where it simply adds its own write load.

First: Quantify, Don’t Panic

Start with the per-device and per-process counters. The goal is to separate genuine application writes from filesystem-level amplification, where one logical write becomes several physical writes.

# device-level: sectors written per second
iostat -x 2 5 | grep -E 'Device|vda|vdb|nvme'

# per-process write throughput (needs sysstat / pidstat)
pidstat -d 2 5

# per cgroup (systemd services)
systemd-cgtop -m

# biggest writers in the last interval
pidstat -d 1 1 | sort -k5 -rn | head

iostat shows what the device is doing; pidstat -d shows who is doing it. If device writes vastly exceed the sum of process writes, the amplification is below the process layer (filesystem, journal, or the hypervisor). If process writes match device writes, you are looking at an application problem.

Common Sources of Amplification

Source	Symptom	Fix
systemd journal without limits	journald writing MB/min steadily	Set SystemMaxUse, Storage=volatile or rate-limit
fsync-heavy databases	Small logical writes, large physical	Tune commit interval, batch fsync
atime updates	Reads cause writes	Mount with noatime
Log rotation storms	Periodic write bursts at fixed times	Stagger cron, tune logrotate size
Snapshots / CoW	Write = old block + new block	Prune snapshots, reconsider CoW
Monitoring agents	Constant small writes to local TSDB	Reduce scrape frequency or store remotely

The Journal Is Usually Guilty

On a quiet VPS, an unbounded journald is one of the most frequent culprits. A chatty unit logs tens of thousands of lines an hour, journald fsyncs them to disk, and the underlying filesystem may write a metadata block for each. Cap it and, if you have remote log shipping, store volatily:

# /etc/systemd/journald.conf
SystemMaxUse=200M
SystemKeepFree=500M
RateLimitIntervalSec=30s
RateLimitBurst=1000

# find the loudest units
journalctl --since '1 hour ago' | awk '{print $5}' | sort | uniq -c | sort -rn | head

Isolate With eBPF When pidstat Isn’t Enough

If process-level accounting is inconclusive, trace the actual write syscalls. biosnoop and biolatency from bcc give per-I/O latency and the owning process without modifying your application.

biosnoop -d vda            # who issues each block write
biolatency -D               # I/O latency distribution
ext4slower 1                # slow ext4 operations with pid
# btrfs variant
btrfsslower 1

Run these for 60 seconds under normal load. A file showing up repeatedly in ext4slower output with high latency and small size is your amplification hot spot; the process id in the output tells you which unit to tune.

Where Monitoring Fits In

Self-hosted monitoring stacks (Prometheus + node_exporter + Grafana, or a lighter pairing like Netdata) are excellent at showing you the shape of the problem over time: the write rate curve, its periodicity, and its correlation with deploys or cron windows. They are much worse on a tiny VPS if you let them store their time-series data locally on the same disk you are trying to protect.

The practical rules: keep retention short (7-15 days) on small instances, use the remote-write feature to push metrics off-box, and never let the monitoring agent’s own local storage become a top-three writer. Check pidstat -d after adding any agent; if it appears in the top writers, reduce its scrape or flush interval before you do anything else.

A Repeatable Diagnostic Order

• Confirm the write rate and its periodicity with iostat -x.
• Attribute it to processes with pidstat -d and systemd-cgtop.
• If attribution fails, trace with biosnoop/ext4slower.
• Check filesystem-level factors: mount options, journald limits, snapshots.
• Only then add or tune monitoring, and verify it is not itself a writer.

Getting the baseline right also influences plan sizing. Sustained writes that keep the disk busy more than a few percent of the time usually mean the workload or the volume is undersized, and the capacity tradeoffs are covered in the cloud VPS benefits overview. If you decided to run this stack yourself rather than have it managed, the operational split is spelled out in the managed versus unmanaged comparison. Fixing the amplification source first is what makes any monitoring worth reading.