Measuring NVMe Write Amplification on a VPS with fio

SSDs write in 4 KiB pages but erase in 256 KiB to 4 MiB blocks. When a block must be rewritten, the controller copies still-valid pages into a fresh block before erasing the old one. That copy traffic is write amplification, and on a VPS it is largely invisible: the guest sees 4 KiB logical writes while the device underneath performs far more. This article shows how to quantify the gap with fio and SMART counters, and which workload patterns drive it up.

The metrics that matter

Write amplification factor (WAF) is host-side bytes written divided by guest-side bytes written. A WAF of 1.0 means perfect mapping; 3.0 means three times the physical wear for the same application traffic. On a VPS you have two ways to observe it: vendor SMART attributes exposed through a paravirtual interface, and latency/IOPS divergence between sequential and random workloads.

Most enterprise NVMe drives report Data Units Written (SMART attribute 241 or the NVMe log page) as a 512,000-byte-unit count from power-on. Sample it before and after a fixed test run and you get host-side bytes directly. Some virtualized drivers do not pass this through; in that case you fall back on the divergence method described below.

Checking what the guest can see

Start with identification. nvme list shows model, namespace size and the driver in use. If the device appears as /dev/sda with a virtio-blk driver rather than nvme0n1, SMART passthrough is usually unavailable and you should rely on the divergence method.

$ sudo nvme list
Node        SN          Model               Namespace  Usage
/dev/nvme0n1 S4EWNX0M…   Samsung PM9A3       1          39.06 GB

$ sudo smartctl -a /dev/nvme0n1 | grep -E 'Data Units Written|Percentage Used|Power On'
Data Units Written:     1,284,236 [657 TB]
Percentage Used:        3%
Power On Hours:         4,211

$ cat /sys/block/nvme0n1/queue/scheduler
none

Record the Data Units Written value (multiply by 512,000 for bytes) and Percentage Used before the test. If the drive reports Percentage Used at 0% after months of heavy use, passthrough is likely broken and the counter is not trustworthy.

Choosing the fio workload

Write amplification is driven by small random writes and by filesystem allocation patterns, so a plain sequential test will not reveal it. Run three workloads against the same file: 4 KiB random writes, 128 KiB random writes, and a mixed 70/30 read/write at 4 KiB. Disable the filesystem page cache in the job so measurements reflect the device, and use --direct=1 with an aligned block size.

# 5 GB test file, pre-created so allocation is not part of the run
$ fio --name=prep --filename=/data/fiotest --size=5G \
    --rw=write --bs=1M --direct=1 --create_on_open=1 --runtime=1 --time_based

$ fio --name=rand4k --filename=/data/fiotest --size=5G \
    --rw=randwrite --bs=4k --iodepth=32 --numjobs=1 \
    --direct=1 --fdatasync=1 --runtime=300 --time_based \
    --group_reporting --output-format=json --output=/tmp/rand4k.json

--fdatasync=1 forces a flush every write, which is what a database does. Without it, the test saturates the drive’s write cache and the numbers flatter the device. Keep runtime fixed at 300 s per workload so the three results are comparable, and run each twice to see the variance before drawing conclusions.

Reading the results

The JSON output carries everything needed: write.iops, write.bw_bytes, and per-latency percentiles. The ratio between sequential 128 KiB bandwidth and random 4 KiB bandwidth scaled to the same byte volume is the practical signal. A drive that sustains 900 MB/s sequential but only 42 MB/s at 4 KiB random is doing roughly 20x the internal work per delivered byte.

$ jq -r '.jobs[0].write | "iops=\(.iops) bw_MB=\(.bw_bytes/1048576|floor) p99_us=\(.clat_ns.percentile["99.000000"]/1000|floor)"' \
    /tmp/rand4k.json
iops=10752 bw_MB=42 p99_us=1841

$ sudo smartctl -A /dev/nvme0n1 | grep 'Data Units Written'
Data Units Written:     1,285,091 [658 TB]

# host-side delta over a 300 s run that wrote 12.6 GB from the guest
# 855 units * 512000 = 437,760,000 bytes  -> sample is too coarse; use a longer run

That last point is important: on this drive one counter unit is 512 KB, so a five-minute test moves the counter by less than a thousand units. Run for at least an hour, or write 50 GB or more, before computing WAF from SMART. Anything shorter is dominated by rounding.

What raises write amplification on a VPS

Overprovisioning is the dominant control. If the namespace occupies the whole raw device with no spare area, the controller has nowhere to place fresh blocks and must erase-and-copy eagerly. Hosts that thin-provision and leave 7–28% unallocated see materially lower WAF. You cannot change this from inside the guest, but you can ask your provider, and you can check Percentage Used growth as a proxy.

Filesystem choices matter next. A filesystem with copy-on-write semantics and frequent snapshots multiplies writes; ext4 with data=ordered is the low-amplification baseline. Record sizes above the device’s physical sector size, aligned partitions, and disabling atime all reduce the number of partial-page writes the controller has to merge.

Mitigations inside the guest

Batch small writes where the application allows it. Database engines already do this with their redo log; application-level logging that fsyncs on every line does not. Set noatime, align partitions to 1 MiB, and avoid swap on NVMe unless you have measured the latency — swap turns random read traffic into random write traffic, which is the worst case for amplification.

Enable TRIM where the provider supports discard. fstrim -av run weekly from a timer tells the controller which blocks are free, which directly reduces copy work during garbage collection. If fstrim returns errors, the virtual block layer is not passing discard through and the drive will degrade faster than its rated endurance suggests.

$ sudo systemctl enable --now fstrim.timer
$ sudo fstrim -av
/data: 34.2 GiB (36708159488 bytes) trimmed on /dev/nvme0n1p1

# verify discard support
$ lsblk -D /dev/nvme0n1
NAME    DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
nvme0n1        0      512B       2G         0

Turning the measurement into a monitoring habit

Track Percentage Used and Data Units Written on a weekly scrape alongside your usual host metrics. A drive climbing more than 2% per month on a workload you consider light is a signal to investigate: either a log file is fsyncing far more than expected, or the host is not overprovisioning and wear rates are inflated for everyone on it.

If you are choosing a plan, ask what the storage tier actually is. Our NVMe IOPS benchmark methodology covers the test sequence for comparing drives on paper, and our VPS plans list the storage type and provisioning per tier so the endurance question can be answered before you buy rather than six months in.

Leave a Reply