{"id":1348,"date":"2026-10-09T22:35:15","date_gmt":"2026-10-09T22:35:15","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1348"},"modified":"2026-10-09T22:35:15","modified_gmt":"2026-10-09T22:35:15","slug":"diagnosing-sustained-disk-write-amplification-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/diagnosing-sustained-disk-write-amplification-vps\/","title":{"rendered":"Diagnosing Sustained Disk-Write Amplification on a VPS (and When to Add Monitoring)"},"content":{"rendered":"\n

A VPS that does nothing but still lights up its disk I\/O counter is a common and frustrating problem. The writes are real, but the process responsible usually is not the one you would guess. This article walks through a systematic method to trace sustained write amplification to its source, and discusses where a self-hosted monitoring and uptime stack helps versus where it simply adds its own write load.<\/p>\n\n\n\n

First: Quantify, Don’t Panic<\/h2>\n\n\n\n

Start with the per-device and per-process counters. The goal is to separate genuine application writes from filesystem-level amplification, where one logical write becomes several physical writes.<\/p>\n\n\n\n

# device-level: sectors written per second\niostat -x 2 5 | grep -E 'Device|vda|vdb|nvme'\n\n# per-process write throughput (needs sysstat \/ pidstat)\npidstat -d 2 5\n\n# per cgroup (systemd services)\nsystemd-cgtop -m\n\n# biggest writers in the last interval\npidstat -d 1 1 | sort -k5 -rn | head<\/code><\/pre>\n\n\n\n
iostat<\/code> shows what the device is doing; pidstat -d<\/code> shows who is doing it. If device writes vastly exceed the sum of process writes, the amplification is below the process layer (filesystem, journal, or the hypervisor). If process writes match device writes, you are looking at an application problem.<\/p>\n\n\n\n
Common Sources of Amplification<\/h2>\n\n\n\n
Source<\/th>Symptom<\/th>Fix<\/th><\/tr><\/thead>
systemd journal without limits<\/td>journald writing MB\/min steadily<\/td>Set SystemMaxUse, Storage=volatile or rate-limit<\/td><\/tr>
fsync-heavy databases<\/td>Small logical writes, large physical<\/td>Tune commit interval, batch fsync<\/td><\/tr>
atime updates<\/td>Reads cause writes<\/td>Mount with noatime<\/td><\/tr>
Log rotation storms<\/td>Periodic write bursts at fixed times<\/td>Stagger cron, tune logrotate size<\/td><\/tr>
Snapshots \/ CoW<\/td>Write = old block + new block<\/td>Prune snapshots, reconsider CoW<\/td><\/tr>
Monitoring agents<\/td>Constant small writes to local TSDB<\/td>Reduce scrape frequency or store remotely<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
The Journal Is Usually Guilty<\/h2>\n\n\n\n
On a quiet VPS, an unbounded journald is one of the most frequent culprits. A chatty unit logs tens of thousands of lines an hour, journald fsyncs them to disk, and the underlying filesystem may write a metadata block for each. Cap it and, if you have remote log shipping, store volatily:<\/p>\n\n\n\n
# \/etc\/systemd\/journald.conf\nSystemMaxUse=200M\nSystemKeepFree=500M\nRateLimitIntervalSec=30s\nRateLimitBurst=1000\n\n# find the loudest units\njournalctl --since '1 hour ago' | awk '{print $5}' | sort | uniq -c | sort -rn | head<\/code><\/pre>\n\n\n\n
Isolate With eBPF When pidstat Isn’t Enough<\/h2>\n\n\n\n
If process-level accounting is inconclusive, trace the actual write syscalls. biosnoop<\/code> and biolatency<\/code> from bcc give per-I\/O latency and the owning process without modifying your application.<\/p>\n\n\n\n
biosnoop -d vda            # who issues each block write\nbiolatency -D               # I\/O latency distribution\next4slower 1                # slow ext4 operations with pid\n# btrfs variant\nbtrfsslower 1<\/code><\/pre>\n\n\n\n
Run these for 60 seconds under normal load. A file showing up repeatedly in ext4slower<\/code> output with high latency and small size is your amplification hot spot; the process id in the output tells you which unit to tune.<\/p>\n\n\n\n
Where Monitoring Fits In<\/h2>\n\n\n\n
Self-hosted monitoring stacks (Prometheus + node_exporter + Grafana, or a lighter pairing like Netdata) are excellent at showing you the shape of the problem over time: the write rate curve, its periodicity, and its correlation with deploys or cron windows. They are much worse on a tiny VPS if you let them store their time-series data locally on the same disk you are trying to protect.<\/p>\n\n\n\n
The practical rules: keep retention short (7-15 days) on small instances, use the remote-write feature to push metrics off-box, and never let the monitoring agent’s own local storage become a top-three writer. Check pidstat -d<\/code> after adding any agent; if it appears in the top writers, reduce its scrape or flush interval before you do anything else.<\/p>\n\n\n\n
A Repeatable Diagnostic Order<\/h2>\n\n\n\n