{"id":1123,"date":"2026-09-13T22:35:41","date_gmt":"2026-09-13T22:35:41","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=1123"},"modified":"2026-09-13T22:35:41","modified_gmt":"2026-09-13T22:35:41","slug":"automated-backup-verification-restore-drills-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/","title":{"rendered":"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">An untested backup is a hypothesis, not a safety net. The failure mode is depressingly common: nightly <code>tar<\/code> or database dumps run for months, exit code zero, and then the day you need them the archive is truncated, the credentials have rotated, or the dump was of the wrong database. This guide builds automated verification \u2014 checksums, restore drills into a scratch area, and alerts \u2014 so you learn your backups are broken on a Tuesday afternoon rather than during an incident.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Verification workloads need disk headroom and enough RAM to start a temporary database. If your instance is tight, size it against the <a href=\"https:\/\/virtualserversvps.com\/\">VPS configurations on our main site<\/a> before automating drills.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>An existing backup routine producing files (dumps, archives, snapshots)<\/li><li>Free disk space of at least 2x the largest backup for a scratch restore<\/li><li><code>sqlite3<\/code>, <code>psql<\/code>, or <code>mysql<\/code> client depending on your stack<\/li><li>A way to send alerts (mail, webhook, or a log monitored by your uptime check)<\/li><li>Root or sudo access and the ability to schedule systemd timers or cron<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1: Make Every Backup Self-Describing<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A backup should carry proof of its own integrity. Write a checksum and a manifest alongside each file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>BACKUP_DIR=\/srv\/backups\nSTAMP=$(date +%Y%m%d-%H%M%S)\nOUT=\"$BACKUP_DIR\/db-$STAMP.sql.gz\"\n\npg_dump -Fc mydb | gzip -9 &gt; \"$OUT\"\nsha256sum \"$OUT\" &gt; \"$OUT.sha256\"\n\ncat &gt; \"$OUT.manifest\" &lt;&lt;EOF\ncreated=$STAMP\nhost=$(hostname -f)\nsource=mydb\nsize_bytes=$(stat -c %s \"$OUT\")\npg_version=$(pg_dump --version)\nEOF\n\n# Fail loudly if the dump is suspiciously small\nSIZE=$(stat -c %s \"$OUT\")\n[ \"$SIZE\" -lt 1048576 ] &amp;&amp; { echo \"ABORT: dump under 1MB\"; exit 1; }<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That size guard alone catches a large fraction of real failures: a dump that suddenly drops from 400MB to 2KB means authentication failed or the schema changed, and it should page you immediately rather than quietly accumulate.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2: Verify Integrity Without a Full Restore<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The cheap check runs every night. It confirms the file is intact and structurally readable without touching your production services.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>LATEST=$(ls -t \/srv\/backups\/db-*.sql.gz | head -1)\n\n# 1. Checksum\nsha256sum -c \"$LATEST.sha256\" || exit 2\n\n# 2. gzip stream integrity\ngzip -t \"$LATEST\" || exit 3\n\n# 3. Archive table of contents (PostgreSQL custom format)\nzcat \"$LATEST\" | pg_restore -l &gt; \/tmp\/bk.toc || exit 4\ngrep -q \"TABLE DATA public orders\" \/tmp\/bk.toc || exit 5\n\n# 4. For MySQL dumps, confirm the header and a known table\n# zcat \"$LATEST\" | head -50 | grep -q \"CREATE TABLE \\`orders\\`\" || exit 5\n\necho \"integrity OK: $LATEST\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Step 3 is the one people skip. A gzip that decompresses cleanly can still contain a dump of an empty database. Checking the table of contents \u2014 and asserting that a known business-critical table is present and non-empty \u2014 is what turns a file check into a real verification.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3: Restore Drill Into an Isolated Scratch Instance<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once a week, restore for real \u2014 but never over production. Run a throwaway database on a non-standard port with its own data directory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>SCRATCH=\/srv\/restore-drill\nDRILL_PORT=55432\nrm -rf \"$SCRATCH\"; mkdir -p \"$SCRATCH\"\nchown postgres:postgres \"$SCRATCH\"\n\nsudo -u postgres \/usr\/lib\/postgresql\/16\/bin\/initdb -D \"$SCRATCH\/data\" &gt;\/dev\/null\n\nsudo -u postgres \/usr\/lib\/postgresql\/16\/bin\/pg_ctl -D \"$SCRATCH\/data\" \\\n  -o \"-p $DRILL_PORT -k $SCRATCH\" -l \"$SCRATCH\/log\" start\nsleep 3\n\nsudo -u postgres createdb -p $DRILL_PORT -h \"$SCRATCH\" drill\nzcat \"$LATEST\" | sudo -u postgres pg_restore -p $DRILL_PORT -h \"$SCRATCH\" -d drill\n\necho \"rows: $(sudo -u postgres psql -p $DRILL_PORT -h $SCRATCH -d drill \\\n  -tAc 'select count(*) from orders')\"\n\nsudo -u postgres \/usr\/lib\/postgresql\/16\/bin\/pg_ctl -D \"$SCRATCH\/data\" stop<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Assert on row counts, not just exit codes. <code>pg_restore<\/code> can return non-zero on harmless warnings, and it can return zero after restoring an empty table. Compare the restored count against the live count within a tolerance band and record the result.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 4: Schedule It and Alert on Failure<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Use a systemd timer so failures are visible in <code>journalctl<\/code> and the unit&#8217;s exit status is recorded.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo tee \/etc\/systemd\/system\/backup-drill.service &gt; \/dev\/null &lt;&lt;'EOF'\n[Unit]\nDescription=Weekly backup restore drill\nOnFailure=alert@%n.service\n\n[Service]\nType=oneshot\nExecStart=\/usr\/local\/bin\/backup-drill.sh\nNice=10\nIOSchedulingClass=idle\nEOF\n\nsudo tee \/etc\/systemd\/system\/backup-drill.timer &gt; \/dev\/null &lt;&lt;'EOF'\n[Unit]\nDescription=Run backup restore drill weekly\n\n[Timer]\nOnCalendar=Sun 04:30\nPersistent=true\nRandomizedDelaySec=900\n\n[Install]\nWantedBy=timers.target\nEOF\n\nsudo systemctl daemon-reload\nsudo systemctl enable --now backup-drill.timer\nsystemctl list-timers | grep backup-drill<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>Persistent=true<\/code> means a drill missed because the VPS was down runs on next boot instead of being silently skipped. <code>RandomizedDelaySec<\/code> avoids every host in your fleet hammering shared storage at the same second.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Verification Step<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Confirm the whole pipeline fails correctly, not just that it succeeds when things go well.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># 1. Run the drill manually and check status\nsudo systemctl start backup-drill.service\nsystemctl status backup-drill.service --no-pager\njournalctl -u backup-drill.service --since \"5 min ago\"\n\n# 2. Negative test: corrupt a copy and confirm detection\ncp \/srv\/backups\/db-*.sql.gz \/tmp\/corrupt.gz\nprintf 'junk' | dd of=\/tmp\/corrupt.gz bs=1 seek=5000 conv=notrunc\ngzip -t \/tmp\/corrupt.gz; echo \"gzip exit (expect non-zero): $?\"\n\n# 3. Confirm the on-failure alert path works\ngrep -q \"ExecStart\" \/etc\/systemd\/system\/backup-drill.service &amp;&amp; echo \"unit OK\"\nsystemctl show backup-drill.service -p OnFailure<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A drill that has never failed a test is not verified. Deliberately breaking a copy and watching the pipeline catch it is the only way to trust the green result.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Troubleshooting<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Symptom<\/th><th>Cause<\/th><th>Fix<\/th><\/tr><\/thead><tbody><tr><td>Drill passes but restore is empty<\/td><td>No row-count assertion<\/td><td>Compare restored vs live counts<\/td><\/tr><tr><td><code>pg_restore<\/code> exits non-zero<\/td><td>Owner\/role missing in scratch instance<\/td><td>Pre-create roles or restore with <code>--no-owner<\/code><\/td><\/tr><tr><td>Scratch instance will not start<\/td><td>Port conflict or missing data dir perms<\/td><td>Use a high port; <code>chown postgres<\/code> the directory<\/td><\/tr><tr><td>Timer never fires<\/td><td><code>OnCalendar<\/code> syntax invalid<\/td><td>Validate with <code>systemd-analyze calendar \"Sun 04:30\"<\/code><\/td><\/tr><tr><td>Alerts silent on failure<\/td><td><code>OnFailure<\/code> unit not defined<\/td><td>Create the alert template unit<\/td><\/tr><tr><td>Disk fills during drill<\/td><td>Scratch not cleaned per run<\/td><td><code>rm -rf<\/code> the scratch dir at script start<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Once this runs for a few weeks you will have a restore time baseline and a failure history \u2014 both essential inputs when you plan a real recovery. Backups stop being a ritual and start being evidence, and the <code>OnFailure<\/code> trail is the audit log that proves it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Automate backup verification on a VPS: checksums, archive content assertions, weekly restore drills into an isolated instance, systemd timer scheduling, and failure alerting.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":1,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1123","post","type-post","status-publish","format-standard","hentry","category-vps-guides-tutorials"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Automated Backup Verification on a VPS: Building Restore Drills That Actually Run - Virtual Servers VPS Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run\" \/>\n<meta property=\"og:description\" content=\"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-13T22:35:41+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/\",\"name\":\"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-09-13T22:35:41+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run - Virtual Servers VPS Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/","og_locale":"en_US","og_type":"article","og_title":"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run","og_description":"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run","og_url":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-09-13T22:35:41+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/","url":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/","name":"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-09-13T22:35:41+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/automated-backup-verification-restore-drills-vps\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Automated Backup Verification on a VPS: Building Restore Drills That Actually Run"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=1123"}],"version-history":[{"count":1,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1123\/revisions"}],"predecessor-version":[{"id":1125,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/1123\/revisions\/1125"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=1123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=1123"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=1123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}