NVMe Wearout Telemetry on a VPS: Reading SMART Attributes to Predict SSD Failure

An NVMe drive fails in a predictable sequence: endurance is consumed, spare blocks are reallocated, the media error rate climbs, and only then does write latency spike and the device drop off the bus. Every stage of that sequence is reported by the drive as a counter, and the counters are readable from inside a KVM guest as long as the controller is passed through or the host exposes the namespace. This guide covers which attributes actually predict failure, what the numeric thresholds mean for a typical VPS workload, and how to wire them into a monitor that alerts before the filesystem goes read-only.

Confirm the device is an NVMe controller with nvme list: a virtio /dev/vda exposes only a synthetic health value, while a namespace under /dev/nvme0 exposes the full log pages. The VPS storage tuning guides at virtualserversvps.com cover the I/O and filesystem side once the controller is confirmed.

The Attributes That Matter and the Ones That Do Not

NVMe SMART exposes 14 counters and five carry real predictive weight: Percentage Used, Available Spare, Media and Data Integrity Errors, Unsafe Shutdowns and the Critical Warning bitmask. Temperature and power-on count are context, not predictors.

Read them with nvme smart-log /dev/nvme0 -o json for machine parsing or smartctl -a /dev/nvme0 for humans. Some virtualised controllers return the log page with Data Units Written zeroed, which makes host-assisted monitoring the only option.

Interpreting Percentage Used Against Real Workloads

Percentage Used is not a linear countdown to failure; it is the drive’s internal estimate of consumed program/erase cycles relative to its rated endurance, and it can jump quickly under sustained heavy writes or creep slowly for years. A consumer 1 TB TLC drive rated at 600 TBW shows 1 percent used per 6 TB written; an enterprise 3.84 TB drive rated at 1 drive write per day for five years shows 1 percent per 7 TB written. At 10 GB per day of write amplification, the consumer drive reaches 100 percent in about 1.6 years.

The second column matters more than the first once the drive is past 80 percent. A drive at 90 percent used with 100 percent spare is consuming its endurance gradually; a drive at 45 percent used with 96 percent spare has already lost 4 percent of its reserve pool, which means blocks are failing and being retired faster than the endurance model predicted. The combination of high percentage used and falling spare is the strongest single failure signal in NVMe telemetry.

Media Errors, Unsafe Shutdowns and the Write Latency Curve

Media and Data Integrity Errors should read zero on a healthy drive. A single increment is an uncorrectable error that was recovered by RAID or the filesystem; three increments in a month on a non-RAID VPS is a reason to migrate, because the drive is now returning bad data rather than merely wearing out. Some consumer controllers report a baseline of one or two for firmware-handled events, so compare against the drive’s own history rather than an absolute zero.

Unsafe Shutdowns counts resets without a clean shutdown; values above 50 usually mean host resets risk an incomplete FTL mapping update. Watch Warning Temperature Time too: excursions above 70 C throttle writes into a visible latency step change.

Capturing a Baseline with smartctl and nvme-cli

Before you can alert on a change you need a baseline. Capture the full log once at provisioning and store it with the hostname and date. The commands below write both the human-readable report and the raw log pages into a directory that can be archived.

d=/var/lib/nvme-baseline/$(hostname)-$(date +%F); mkdir -p $d
smartctl -a /dev/nvme0 > $d/smart.txt
nvme smart-log /dev/nvme0 -o json > $d/smart.json
nvme id-ctrl /dev/nvme0 -o json > $d/idctrl.json
nvme error-log /dev/nvme0 > $d/errors.txt

id-ctrl records model, firmware revision and rated endurance; error-log lists the last 64 status codes and should be empty on a healthy drive. Re-diff these files after a firmware update, because counters can be reset or reinterpreted.

Automating Collection into node_exporter or a Cron Probe

The node_exporter textfile collector is the shortest path to Prometheus: a cron script writes a metrics file every five minutes, and node_exporter reads it via --collector.textfile.directory=/var/lib/node_exporter/textfile.

#!/bin/bash
out=/var/lib/node_exporter/textfile/nvme.prom
tmp=$out.$$
{
  j=$(nvme smart-log /dev/nvme0 -o json)
  echo "nvme_percentage_used $(echo "$j" | jq '.percentage_used')"
  echo "nvme_available_spare $(echo "$j" | jq '.avail_spare')"
  echo "nvme_media_errors $(echo "$j" | jq '.media_errors')"
  echo "nvme_unsafe_shutdowns $(echo "$j" | jq '.unsafe_shutdowns')"
  echo "nvme_data_units_written $(echo "$j" | jq '.data_units_written')"
  echo "nvme_critical_warning $(echo "$j" | jq '.critical_warning')"
} > $tmp && mv $tmp $out

Set alert rules on the derivative, not just the level. A rate of nvme_percentage_used above 1.5 per week means the drive will hit 100 percent within a year and the workload needs write reduction. Any change in nvme_critical_warning from zero triggers a page, and nvme_available_spare < 10 triggers an immediate migrate. Keep a recording rule for the 30-day delta so a sudden acceleration is visible even when the absolute value is low.

Reducing Write Amplification Before It Kills the Drive

Wear equals host writes times amplification, and on a small VPS the amplification is usually self-inflicted. Check findmnt -no OPTIONS / for noatime, confirm /sys/block/nvme0n1/queue/scheduler reads none, and verify discard reaches the device with lsblk --discard plus a weekly fstrim.timer.

Journald is the other consumer: journalctl --disk-usage shows consumption, and SystemMaxUse=200M with Compress=yes caps it. For databases, innodb_flush_log_at_trx_commit=2 cuts small writes about 40 percent for up to one second of durability.

Reading the Telemetry When smartctl Returns Nothing

Many KVM providers expose the disk as virtio and the guest sees no SMART data at all. Before assuming the drive is opaque, try smartctl -d scsi -a /dev/vda and nvme list-subsys; some controllers pass through a reduced log set with only the critical warning bit. If both return nothing, the guest cannot monitor the device and the check has to happen host-side.

In that case, build an application-level proxy metric: track the 99th percentile fsync latency from the database, since a worn drive shows it first. In PostgreSQL, pg_stat_io on version 16 and later reports write_time and fsync_time per backend type, and a rising fsync time with flat query volume is the same signal as a falling available spare. Retain those samples for comparison across drive generations.

When to Migrate and How to Verify the New Drive

Migrate when two of these hold: used above 80 percent, spare below 99, rising media errors, or QD1 write latency above 400 microseconds where provisioning measured 120. Snapshot with fsfreeze --freeze /data around the copy and unfreeze after, or the journal will be inconsistent.

On the new host, capture the same baseline files and compare model, firmware and rated endurance. Run fio --name=w --rw=write --bs=128k --size=8G --direct=1 to confirm throughput, and log the starting percentage used so the wear rate is computable without waiting a month.

Two weeks after cutover, compare the new drive’s percentage used against the old one at the same data unit count; a materially higher figure means the new volume has worse write amplification and the workload needs tuning. Archive the baseline directory with the host decommission record so a repeated failure has a reference point. Hosts where the provider exposes the controller are the only ones where this telemetry exists, and the NVMe-backed VPS plans compared at virtualserversvps.com list the drive class per tier.