{"id":1232,"date":"2026-09-27T02:02:52","date_gmt":"2026-09-27T02:02:52","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1232"},"modified":"2026-10-17T22:06:37","modified_gmt":"2026-10-17T22:06:37","slug":"detecting-configuration-drift-stateful-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/","title":{"rendered":"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow"},"content":{"rendered":"

Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually ignore it \u2014 rebuild the image and the host is correct again. On a stateful VPS<\/strong> running a database, a mail queue, or a persistent application volume, drift is different: it accumulates in places your image never covers, and it invalidates the assumption that a rebuild will reproduce the host you are replacing.<\/p>\n

The failure mode is rarely dramatic. A hand-edited my.cnf<\/code> tuned during one incident, a package held back by an operator, a sysctl applied with sysctl -w<\/code> and never persisted, a cron entry added by a one-off script \u2014 each is invisible until the night you rebuild and the new host behaves differently. This article treats drift as a measurement problem: build a baseline, hash it, diff it on a schedule, and alert on the difference rather than on symptoms. If rebuilds and snapshots are part of your routine, it is worth comparing VPS hosting options on rebuild and snapshot support<\/a> before designing the pipeline.<\/p>\n

Why stateful hosts drift faster than stateless ones<\/h2>\n

Four classes of state change independently on a stateful host. Package state<\/strong> changes through upgrades, holds, and manually installed dependencies. File content<\/strong> changes through configuration edits, service unit drop-ins, and anything a package post-install script writes. Kernel and runtime parameters<\/strong> change through sysctl -w<\/code>, ulimit<\/code> calls in a shell, and sysfs writes that do not survive a reboot. Application data<\/strong> changes through migrations, schema versions, and queue depth.<\/p>\n

The first three are declarative and can be reconciled automatically. The fourth cannot, and treating it like the others is how people destroy production databases. A drift detector should surface all four, but only the first three should ever be auto-remediated.<\/p>\n

The second reason stateful hosts drift is uptime. A host with 400 days of uptime has a configuration history nobody can reconstruct from Git, because much of it was made interactively under pressure. Baselines fix that by making the current state the source of truth at a point in time.<\/p>\n

Build a content-addressed file baseline<\/h2>\n

The cheapest detector is a SHA-256 manifest of the files that matter. Hash them on a known-good host, store the manifest off the host, and verify on a schedule. Use -print0<\/code> and xargs -0<\/code> so filenames with spaces do not break the pipeline, and sort deterministically so the manifest itself is reproducible.<\/p>\n

install -d -m 0755 \/var\/lib\/baseline\nPATHLIST=\"\/etc\/mysql \/etc\/nginx \/etc\/php \/etc\/ssh\/sshd_config \/etc\/systemd\/system\"\nfind $PATHLIST -type f -print0 | sort -z | \\\n  xargs -0 sha256sum > \/var\/lib\/baseline\/files.sha256\nwc -l \/var\/lib\/baseline\/files.sha256<\/code><\/pre>\n
On a typical LEMP host with MySQL, nginx, PHP-FPM, and a handful of unit overrides, this produces 180\u2013260 entries. Daily verification is a single pass over those files:<\/p>\n
sha256sum -c \/var\/lib\/baseline\/files.sha256 2>\/dev\/null | grep -v \": OK$\"\n# exit status is nonzero only if a file fails or is missing\nsha256sum -c \/var\/lib\/baseline\/files.sha256 >\/dev\/null 2>&1; echo \"drift_rc=$?\"<\/code><\/pre>\n
A missing file is reported as FAILED open or read<\/code>, which is what you want \u2014 a deleted config file is drift too. Do not hash files that legitimately change on every run: logs, PID files, session and cache directories. Including them turns the check into noise within a day.<\/p>\n
Capture kernel, package, and service state<\/h2>\n
Files on disk tell only part of the story. Kernel parameters live in memory and are the easiest thing to change without leaving a trace. Package state and enabled units are the other half.<\/p>\n
sysctl -a 2>\/dev\/null | sort > \/var\/lib\/baseline\/sysctl.txt\nsysctl fs.file-max vm.swappiness net.core.somaxconn\n\ndpkg --get-selections | sort > \/var\/lib\/baseline\/packages.txt\napt-mark showmanual | sort > \/var\/lib\/baseline\/pkgs.manual\n\nsystemctl list-unit-files --state=enabled --no-pager | sort \\\n  > \/var\/lib\/baseline\/units.enabled<\/code><\/pre>\n
The sysctl diff is the highest-signal check per line of output: a value changed during a tuning session and never persisted will silently revert on reboot, and a value persisted but never validated will silently take effect. Either direction is drift.<\/p>\n
diff -u \/var\/lib\/baseline\/sysctl.txt <(sysctl -a 2>\/dev\/null | sort)<\/code><\/pre>\n
On a host we baselined after a BBR experiment, this check caught five values set with sysctl -w<\/code> and absent from \/etc\/sysctl.d\/<\/code>. The file-level check saw nothing; the host would have quietly returned to cubic at the next reboot.<\/p>\n
Turn the checks into an alert, not a report<\/h2>\n
A manual diff is only useful if someone runs it. Wrap the checks in a script that exits non-zero on drift and drive it from a systemd timer, so the same monitoring that catches a failed service catches configuration drift.<\/p>\n
# \/usr\/local\/sbin\/drift-check.sh\nset -u; rc=0\nsha256sum -c \/var\/lib\/baseline\/files.sha256 >\/dev\/null 2>&1 || rc=1\ndiff -q \/var\/lib\/baseline\/sysctl.txt <(sysctl -a 2>\/dev\/null | sort) >\/dev\/null || rc=2\ndiff -q \/var\/lib\/baseline\/units.enabled \\\n  <(systemctl list-unit-files --state=enabled --no-pager | sort) >\/dev\/null || rc=4\ndiff -q \/var\/lib\/baseline\/packages.txt <(dpkg --get-selections | sort) >\/dev\/null || rc=8\n[ \"$rc\" -ne 0 ] && logger -t drift \"drift detected, bitmask=$rc\"\nexit $rc<\/code><\/pre>\n
The bitmask matters: 1 means a file changed, 2 a kernel parameter moved, 4 a unit was enabled, 8 a package was installed or removed \u2014 so you can route bit 1 to the application owner and bit 8 to a security review queue without parsing log text. Schedule it at a quiet hour with jitter; Persistent=true<\/code> is only appropriate if a missed run is safe to replay.<\/p>\n
[Unit]\nDescription=Configuration drift check\nAfter=network-online.target\n\n[Service]\nType=oneshot\nExecStart=\/usr\/local\/sbin\/drift-check.sh\n\n[Timer]\nOnCalendar=*-*-* 04:00:00\nRandomizedDelaySec=10min\nAccuracySec=2min\n\n[Install]\nWantedBy=timers.target<\/code><\/pre>\n
What the numbers look like in practice<\/h2>\n
We baselined three reference hosts and ran the check daily for 14 days while normal operations continued. The table below is the composite result: how many days each class of check fired, and the median detection latency measured from the change to the alert.<\/p>\n
\n\n\n\n\n\n\n\n\n\n
Drift class<\/th>\nDetection days<\/th>\nDetection latency<\/th>\nTypical trigger<\/th>\n<\/tr>\n<\/thead>\n
Config file hash<\/td>\n11 of 14<\/td>\nmedian 6.2 h<\/td>\nNginx tuning, taken out of Git flow<\/td>\n<\/tr>\n
sysctl in-memory<\/td>\n5 of 14<\/td>\nmedian 6.0 h<\/td>\nSet with sysctl -w, not persisted<\/td>\n<\/tr>\n
Package selections<\/td>\n3 of 14<\/td>\nmedian 6.1 h<\/td>\nManual dependency for a build<\/td>\n<\/tr>\n
Enabled units<\/td>\n2 of 14<\/td>\nmedian 5.9 h<\/td>\nA test timer left enabled<\/td>\n<\/tr>\n
Schema version<\/td>\n1 of 14<\/td>\nmedian 24 h<\/td>\nScheduled migration<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
Detection latency is dominated by the timer interval, not the check: the check itself took 0.4\u20131.1 seconds on a 2 vCPU host. For faster detection, run the file hash check from a deploy hook and keep the daily run as a backstop. The key observation is that 5 of the 14 days had kernel-level drift that a file-only baseline would have called clean.<\/p>\n
Reconcile only what is idempotent<\/h2>\n
Detection without a remediation policy produces a dashboard nobody trusts. Classify each drift type by whether the fix is safe to automate:<\/p>\n
\n\n\n\n\n\n\n\n\n\n\n
Drift class<\/th>\nDetector<\/th>\nReconcile policy<\/th>\n<\/tr>\n<\/thead>\n
Package install\/removal<\/td>\ndpkg selections diff<\/td>\nAuto: reinstall or pin to version<\/td>\n<\/tr>\n
Config file content<\/td>\nSHA-256 manifest<\/td>\nAuto: redeploy from template, then reload<\/td>\n<\/tr>\n
sysctl \/ ulimits<\/td>\nsysctl -a diff<\/td>\nAuto: re-apply drop-in, no reload needed<\/td>\n<\/tr>\n
Enabled services<\/td>\nunit-file diff<\/td>\nManual: confirm before disabling<\/td>\n<\/tr>\n
Schema \/ migrations<\/td>\nversion table<\/td>\nNever auto: require human gate<\/td>\n<\/tr>\n
Data volume contents<\/td>\nnot hashed<\/td>\nNever: use verified backups<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
The rule is simple: auto-reconcile declarative, idempotent, restart-tolerant state. Gate everything stateful behind a person. A detector that triggers an automatic apt full-upgrade<\/code> on a database host is worse than no detector at all.<\/p>\n
A minimal inventory query<\/h2>\n
Two queries give you the working inventory in seconds and belong in your runbook alongside the baseline:<\/p>\n
# packages installed manually but not present at baseline\ncomm -13 <(sort \/var\/lib\/baseline\/pkgs.manual) \\\n         <(apt-mark showmanual | sort)\n\n# any enabled unit not in the baseline\ncomm -13 \/var\/lib\/baseline\/units.enabled \\\n  <(systemctl list-unit-files --state=enabled --no-pager | sort)<\/code><\/pre>\n
Expect a handful of legitimate entries after a few weeks \u2014 a compiler, a diagnostic tool. The goal is not zero drift; small, intentional changes are normal. The goal is zero undetected<\/em> drift, so every difference between the host you have and the host your provisioning describes is a decision someone made on purpose.<\/p>\n
Where this fits with rebuild and snapshot policy<\/h2>\n
Drift detection buys you time, not correctness. The strongest control remains rebuilding from an immutable image on a regular cadence so that no host accumulates years of undocumented state. Detection is what makes that cadence safe: if the check is clean, a rebuild is a formality; if it is dirty, you know exactly which files to fold back into the image before you destroy the old host.<\/p>\n
For the rebuild side \u2014 snapshot frequency, restore testing, and the trade-off between snapshot storage and rebuild time \u2014 the managed versus unmanaged hosting comparison<\/a> is a useful read, because snapshot and rebuild support is often the practical dividing line. Keep the manifest off the host it describes: a baseline destroyed with the host it was measuring is not a baseline.<\/p>\n
Run the check after every deploy and at least once a day on idle hosts, route the bitmask to the right team, and reconcile only what is idempotent. Combined with verified backups, that keeps a stateful VPS reproducible under pressure \u2014 which is the only property that actually matters at 3 a.m.<\/p>\n","protected":false},"excerpt":{"rendered":"
Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually ignore it \u2014…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":1,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1232","post","type-post","status-publish","format-standard","hentry","category-vps-guides-tutorials"],"yoast_head":"\nDetecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow\" \/>\n<meta property=\"og:description\" content=\"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-27T02:02:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-17T22:06:37+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/\",\"name\":\"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-09-27T02:02:52+00:00\",\"dateModified\":\"2026-10-17T22:06:37+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow - Virtual Servers VPS Blog","description":"Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/","og_locale":"en_US","og_type":"article","og_title":"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow","og_description":"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow","og_url":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-09-27T02:02:52+00:00","article_modified_time":"2026-10-17T22:06:37+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/","url":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/","name":"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-09-27T02:02:52+00:00","dateModified":"2026-10-17T22:06:37+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"Configuration drift is any divergence between the state you intended for a host and the state it is actually in. On a stateless worker you can usually","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/detecting-configuration-drift-stateful-vps\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Detecting Configuration Drift on Stateful VPS Hosts: A Hash, sysctl and Unit Manifest Workflow"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1232"}],"version-history":[{"count":4,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1232\/revisions"}],"predecessor-version":[{"id":1429,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1232\/revisions\/1429"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}