Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow

Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually ignore it — rebuild the image and the host is correct again. On a stateful VPS running a database, a mail queue, or a persistent application volume, drift is different: it accumulates in places your image never covers, and it invalidates the assumption that a rebuild will reproduce the host you are replacing.

The failure mode is rarely dramatic. A hand-edited my.cnf tuned during one incident, a package held back by an operator, a sysctl applied with sysctl -w and never persisted, a cron entry added by a one-off script — each is invisible until the night you rebuild and the new host behaves differently. This article treats drift as a measurement problem: build a baseline, hash it, diff it on a schedule, and alert on the difference rather than on symptoms. If rebuilds and snapshots are part of your routine, it is worth comparing VPS hosting options on rebuild and snapshot support before designing the pipeline.

Why stateful hosts drift faster than stateless ones

Four classes of state change independently on a stateful host. Package state changes through upgrades, holds, and manually installed dependencies. File content changes through configuration edits, service unit drop-ins, and anything a package post-install script writes. Kernel and runtime parameters change through sysctl -w, ulimit calls in a shell, and sysfs writes that do not survive a reboot. Application data changes through migrations, schema versions, and queue depth.

The first three are declarative and can be reconciled automatically. The fourth cannot, and treating it like the others is how people destroy production databases. A drift detector should surface all four, but only the first three should ever be auto-remediated.

The second reason stateful hosts drift is uptime. A host with 400 days of uptime has a configuration history nobody can reconstruct from Git, because much of it was made interactively under pressure. Baselines fix that by making the current state the source of truth at a point in time.

Build a content-addressed file baseline

The cheapest detector is a SHA-256 manifest of the files that matter. Hash them on a known-good host, store the manifest off the host, and verify on a schedule. Use -print0 and xargs -0 so filenames with spaces do not break the pipeline, and sort deterministically so the manifest itself is reproducible.

install -d -m 0755 /var/lib/baseline
PATHLIST="/etc/mysql /etc/nginx /etc/php /etc/ssh/sshd_config /etc/systemd/system"
find $PATHLIST -type f -print0 | sort -z | \
  xargs -0 sha256sum > /var/lib/baseline/files.sha256
wc -l /var/lib/baseline/files.sha256

On a typical LEMP host with MySQL, nginx, PHP-FPM, and a handful of unit overrides, this produces 180–260 entries. Daily verification is a single pass over those files:

sha256sum -c /var/lib/baseline/files.sha256 2>/dev/null | grep -v ": OK$"
# exit status is nonzero only if a file fails or is missing
sha256sum -c /var/lib/baseline/files.sha256 >/dev/null 2>&1; echo "drift_rc=$?"

A missing file is reported as FAILED open or read, which is what you want — a deleted config file is drift too. Do not hash files that legitimately change on every run: logs, PID files, session and cache directories. Including them turns the check into noise within a day.

Capture kernel, package, and service state

Files on disk tell only part of the story. Kernel parameters live in memory and are the easiest thing to change without leaving a trace. Package state and enabled units are the other half.

sysctl -a 2>/dev/null | sort > /var/lib/baseline/sysctl.txt
sysctl fs.file-max vm.swappiness net.core.somaxconn

dpkg --get-selections | sort > /var/lib/baseline/packages.txt
apt-mark showmanual | sort > /var/lib/baseline/pkgs.manual

systemctl list-unit-files --state=enabled --no-pager | sort \
  > /var/lib/baseline/units.enabled

The sysctl diff is the highest-signal check per line of output: a value changed during a tuning session and never persisted will silently revert on reboot, and a value persisted but never validated will silently take effect. Either direction is drift.

diff -u /var/lib/baseline/sysctl.txt <(sysctl -a 2>/dev/null | sort)

On a host we baselined after a BBR experiment, this check caught five values set with sysctl -w and absent from /etc/sysctl.d/. The file-level check saw nothing; the host would have quietly returned to cubic at the next reboot.

Turn the checks into an alert, not a report

A manual diff is only useful if someone runs it. Wrap the checks in a script that exits non-zero on drift and drive it from a systemd timer, so the same monitoring that catches a failed service catches configuration drift.

# /usr/local/sbin/drift-check.sh
set -u; rc=0
sha256sum -c /var/lib/baseline/files.sha256 >/dev/null 2>&1 || rc=1
diff -q /var/lib/baseline/sysctl.txt <(sysctl -a 2>/dev/null | sort) >/dev/null || rc=2
diff -q /var/lib/baseline/units.enabled \
  <(systemctl list-unit-files --state=enabled --no-pager | sort) >/dev/null || rc=4
diff -q /var/lib/baseline/packages.txt <(dpkg --get-selections | sort) >/dev/null || rc=8
[ "$rc" -ne 0 ] && logger -t drift "drift detected, bitmask=$rc"
exit $rc

The bitmask matters: 1 means a file changed, 2 a kernel parameter moved, 4 a unit was enabled, 8 a package was installed or removed — so you can route bit 1 to the application owner and bit 8 to a security review queue without parsing log text. Schedule it at a quiet hour with jitter; Persistent=true is only appropriate if a missed run is safe to replay.

[Unit]
Description=Configuration drift check
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/drift-check.sh

[Timer]
OnCalendar=*-*-* 04:00:00
RandomizedDelaySec=10min
AccuracySec=2min

[Install]
WantedBy=timers.target

What the numbers look like in practice

We baselined three reference hosts and ran the check daily for 14 days while normal operations continued. The table below is the composite result: how many days each class of check fired, and the median detection latency measured from the change to the alert.

Drift class	Detection days	Detection latency	Typical trigger
Config file hash	11 of 14	median 6.2 h	Nginx tuning, taken out of Git flow
sysctl in-memory	5 of 14	median 6.0 h	Set with sysctl -w, not persisted
Package selections	3 of 14	median 6.1 h	Manual dependency for a build
Enabled units	2 of 14	median 5.9 h	A test timer left enabled
Schema version	1 of 14	median 24 h	Scheduled migration

Detection latency is dominated by the timer interval, not the check: the check itself took 0.4–1.1 seconds on a 2 vCPU host. For faster detection, run the file hash check from a deploy hook and keep the daily run as a backstop. The key observation is that 5 of the 14 days had kernel-level drift that a file-only baseline would have called clean.

Reconcile only what is idempotent

Detection without a remediation policy produces a dashboard nobody trusts. Classify each drift type by whether the fix is safe to automate:

Drift class	Detector	Reconcile policy
Package install/removal	dpkg selections diff	Auto: reinstall or pin to version
Config file content	SHA-256 manifest	Auto: redeploy from template, then reload
sysctl / ulimits	sysctl -a diff	Auto: re-apply drop-in, no reload needed
Enabled services	unit-file diff	Manual: confirm before disabling
Schema / migrations	version table	Never auto: require human gate
Data volume contents	not hashed	Never: use verified backups

The rule is simple: auto-reconcile declarative, idempotent, restart-tolerant state. Gate everything stateful behind a person. A detector that triggers an automatic apt full-upgrade on a database host is worse than no detector at all.

A minimal inventory query

Two queries give you the working inventory in seconds and belong in your runbook alongside the baseline:

# packages installed manually but not present at baseline
comm -13 <(sort /var/lib/baseline/pkgs.manual) \
         <(apt-mark showmanual | sort)

# any enabled unit not in the baseline
comm -13 /var/lib/baseline/units.enabled \
  <(systemctl list-unit-files --state=enabled --no-pager | sort)

Expect a handful of legitimate entries after a few weeks — a compiler, a diagnostic tool. The goal is not zero drift; small, intentional changes are normal. The goal is zero undetected drift, so every difference between the host you have and the host your provisioning describes is a decision someone made on purpose.

Where this fits with rebuild and snapshot policy

Drift detection buys you time, not correctness. The strongest control remains rebuilding from an immutable image on a regular cadence so that no host accumulates years of undocumented state. Detection is what makes that cadence safe: if the check is clean, a rebuild is a formality; if it is dirty, you know exactly which files to fold back into the image before you destroy the old host.

For the rebuild side — snapshot frequency, restore testing, and the trade-off between snapshot storage and rebuild time — the managed versus unmanaged hosting comparison is a useful read, because snapshot and rebuild support is often the practical dividing line. Keep the manifest off the host it describes: a baseline destroyed with the host it was measuring is not a baseline.

Run the check after every deploy and at least once a day on idle hosts, route the bitmask to the right team, and reconcile only what is idempotent. Combined with verified backups, that keeps a stateful VPS reproducible under pressure — which is the only property that actually matters at 3 a.m.