{"id":840,"date":"2026-08-10T23:58:39","date_gmt":"2026-08-10T23:58:39","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=840"},"modified":"2026-10-20T22:04:14","modified_gmt":"2026-10-20T22:04:14","slug":"automated-security-updates-unattended-upgrades-vps","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/","title":{"rendered":"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification"},"content":{"rendered":"<div style=\"padding:20px\">\n<p>Unattended updates fail quietly. The service stays &#8220;active&#8221;, the timer fires every day, and nothing gets installed \u2014 because a package is on hold, because a repository is unreachable, or because a previous run left the package database locked. On a single instance that you check weekly this is survivable; across a fleet it means patch latency drifts from days to months without anyone noticing. This guide covers the failure modes that matter, how to detect each one from logs and state files, and how to add the verification step that most setups are missing. It applies to any Linux instance, from <a href=\"https:\/\/virtualserversvps.com\/\">a small single-purpose VPS<\/a> up to a fleet of fleet-managed nodes.<\/p>\n<h2>How the Debian\/Ubuntu Stack Fits Together<\/h2>\n<p>There are three moving pieces, and confusing them is the root cause of most &#8220;why didn&#8217;t my server update&#8221; incidents:<\/p>\n<ul>\n<li><strong>apt-daily.timer<\/strong> \u2014 runs <code>apt-get update<\/code> twice a day, refreshing the package lists.<\/li>\n<li><strong>apt-daily-upgrade.timer<\/strong> \u2014 runs <code>unattended-upgrade<\/code>, which installs from the refreshed lists.<\/li>\n<li><strong>unattended-upgrades.service<\/strong> \u2014 the actual worker, subject to systemd sandboxing restrictions.<\/li>\n<\/ul>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;systemctl list-timers 'apt*' --all\nsystemctl status unattended-upgrades.service --no-pager\ncat \/etc\/apt\/apt.conf.d\/20auto-upgrades&lt;\/code&gt;<\/code><\/pre>\n<p>If <code>20auto-upgrades<\/code> shows both values at <code>0<\/code>, nothing will ever run regardless of what <code>50unattended-upgrades<\/code> says. That is the first check, and it is the most common single-line cause of a stale fleet.<\/p>\n<h2>Failure Mode 1: Held Packages<\/h2>\n<p><code>apt-mark hold<\/code> is sticky and easy to forget. A held package blocks that package only, but a held kernel meta-package (<code>linux-image-generic<\/code>) silently stops all kernel security updates while everything else keeps flowing \u2014 the fleet looks healthy and is missing the patches that matter most.<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;apt-mark showhold\napt-mark showmanual | head -40\ngrep -c 'Packages kept back' \/var\/log\/unattended-upgrades\/unattended-upgrades.log&lt;\/code&gt;<\/code><\/pre>\n<p>Audit holds as part of the same routine that reviews firewall rules. Every hold should have an owner and an expiry date, or it should be removed.<\/p>\n<h2>Failure Mode 2: Repository and Network Failures<\/h2>\n<p>If <code>apt-get update<\/code> fails, unattended-upgrade does nothing and exits 0 in many configurations. That exit code is the trap: your monitoring sees success. Check for stale package lists by timestamp rather than by exit status:<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;stat -c '%y %n' \/var\/lib\/apt\/lists\/*Release | sort | head -5\ndate -u\ngrep -iE 'Err:|W:|Failed to fetch' \/var\/log\/apt\/term.log | tail -20&lt;\/code&gt;<\/code><\/pre>\n<p>A <code>Release<\/code> file older than 48 hours means the daily update has not completed. Common causes on a VPS: an outbound firewall rule added for a different purpose, a proxy environment variable set in one shell but not in systemd&#8217;s environment, or a mirror that no longer carries the release (EOL distributions are the classic case). Verify the proxy is visible to the service, not just to your SSH session:<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;systemctl show apt-daily.service -p Environment\nsystemctl show unattended-upgrades.service -p Environment&lt;\/code&gt;<\/code><\/pre>\n<h2>Failure Mode 3: Sandbox Restrictions Blocking the Worker<\/h2>\n<p>The packaged <code>unattended-upgrades.service<\/code> runs with <code>ProtectSystem<\/code>, <code>PrivateTmp<\/code>, restricted address families and a read-only view of parts of the filesystem. Hooks that write outside the permitted paths \u2014 a custom post-invoke script, a package that stages files in <code>\/var\/tmp<\/code> \u2014 fail with permission errors that appear only in the journal.<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;journalctl -u unattended-upgrades --since '7 days ago' | grep -iE 'permission|read-only|denied'&lt;\/code&gt;<\/code><\/pre>\n<p>The correct fix is usually to remove the offending hook, not to weaken the sandbox. If a hook genuinely needs to write, move it into a separate systemd unit triggered by <code>apt-daily-upgrade.service<\/code> completing.<\/p>\n<h2>Failure Mode 4: Deferred Reboots Piling Up<\/h2>\n<p>Kernel, glibc and systemd updates do not take effect until reboot. The mechanism is <code>\/var\/run\/reboot-required<\/code> plus the <code>reboot-required.pkgs<\/code> list:<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;cat \/var\/run\/reboot-required 2&gt;\/dev\/null\ncat \/var\/run\/reboot-required.pkgs 2&gt;\/dev\/null\nuname -r\nls -1 \/boot\/vmlinuz-* | sort -V | tail -3&lt;\/code&gt;<\/code><\/pre>\n<p>If <code>uname -r<\/code> is several kernels behind the newest installed image, you are patching on disk and not in memory. This matters because the kernel is where most remotely exploitable privilege-escalation fixes land. A reboot policy is therefore part of the patch policy:<\/p>\n<ul>\n<li>Reboot at a defined maintenance window, not immediately after patching \u2014 unless the CVE is being actively exploited.<\/li>\n<li>Notify before rebooting and log the event with the package list that triggered it.<\/li>\n<li>If you cannot reboot, live-patch the kernel (<code>kpatch<\/code> or <code>kernel-livepatch<\/code>) \u2014 but live patching only covers the specific CVEs shipped in the patch stream, never a general upgrade.<\/li>\n<\/ul>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;# example: reboot only in a window, and only when required\ncat &gt;\/usr\/local\/sbin\/patch-reboot-check &lt;&lt;'EOF'\n#!\/bin\/sh\n[ -f \/var\/run\/reboot-required ] || exit 0\nlogger -t patch-reboot \"reboot required: $(tr '\\n' ' ' &lt; \/var\/run\/reboot-required.pkgs)\"\n\/sbin\/shutdown -r +10 \"Scheduled patching reboot\"\nEOF\nchmod +x \/usr\/local\/sbin\/patch-reboot-check&lt;\/code&gt;<\/code><\/pre>\n<h2>Failure Mode 5: The Locked dpkg Database<\/h2>\n<p>Two updaters running simultaneously \u2014 unattended-upgrades and an interactive <code>apt upgrade<\/code>, or a configuration-management run \u2014 cause the second to fail with &#8220;Could not get lock \/var\/lib\/dpkg\/lock-frontend&#8221;. Over SSH this is obvious; under a cron-based tool it becomes a silent skip.<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;lsof \/var\/lib\/dpkg\/lock-frontend\nfuser -v \/var\/lib\/dpkg\/lock 2&gt;\/dev\/null&lt;\/code&gt;<\/code><\/pre>\n<p>Ensure exactly one updater is in charge. If configuration management also patches, disable unattended-upgrades rather than letting both race. Check with <code>ps aux | grep -E 'apt|dpkg'<\/code> at a time when a run should be in progress.<\/p>\n<h2>Verification: Prove Patches Actually Landed<\/h2>\n<p>Exit codes are not evidence. Three checks together give a real answer:<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;# 1. what the last run did\ntail -30 \/var\/log\/unattended-upgrades\/unattended-upgrades.log\n\n# 2. what is pending right now\napt-get -s upgrade | grep -c '^Inst'\n\n# 3. what still has known CVEs\napt-get -s dist-upgrade | grep '^Inst' | grep -i security&lt;\/code&gt;<\/code><\/pre>\n<p>Then track the one number that matters: days since the last successful security install. A simple probe writes it into a file your monitoring already scrapes:<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;#!\/bin\/sh\n# write to a textfile endpoint if using node_exporter\nlast=$(grep 'All upgrades installed' \/var\/log\/unattended-upgrades\/unattended-upgrades.log \\\n       | tail -1 | awk '{print $1}')\nif [ -n \"$last\" ]; then\n  ts=$(date -d \"$last\" +%s 2&gt;\/dev\/null)\n  [ -n \"$ts\" ] &amp;&amp; echo \"$(( ($(date +%s) - ts) \/ 86400 ))\" &gt; \/var\/lib\/node_exporter\/days_since_patch.prom\nfi\ngrep -i security \/var\/log\/unattended-upgrades\/unattended-upgrades.log | wc -l &gt; \/var\/lib\/node_exporter\/security_pkgs_patched.prom&lt;\/code&gt;<\/code><\/pre>\n<p>Alert when the day count exceeds your patch SLA. This single metric catches every failure mode above, because all of them end in the same observable state: no successful security installs for an unusual length of time.<\/p>\n<h2>Mail and Webhook Notifications<\/h2>\n<p>Local mail from a VPS is usually rejected or dropped. Point it at a relay or replace it with a webhook:<\/p>\n<pre class=\"wp-block-code\"><code>&lt;code&gt;Unattended-Upgrade::Mail \"ops@example.com\";\nUnattended-Upgrade::MailReport \"on-change\";&lt;\/code&gt;<\/code><\/pre>\n<p>Set <code>on-change<\/code> rather than <code>always<\/code> if you route notifications to a human; <code>always<\/code> generates noise that trains people to ignore it. For automated handling, a webhook hook in <code>\/etc\/apt\/apt.conf.d\/99webhook<\/code> posting to your incident tool is more reliable than SMTP because it fails loudly.<\/p>\n<h2>A Minimal Weekly Audit<\/h2>\n<ol>\n<li><code>systemctl list-timers 'apt*'<\/code> \u2014 confirm both timers last fired within 24 hours.<\/li>\n<li>Check the newest <code>Release<\/code> file is under 48 hours old.<\/li>\n<li><code>apt-mark showhold<\/code> \u2014 empty or documented.<\/li>\n<li><code>apt-get -s upgrade | grep -c '^Inst'<\/code> \u2014 zero, or explainable.<\/li>\n<li><code>uname -r<\/code> against the installed kernel list.<\/li>\n<li>The days-since-patch metric within SLA.<\/li>\n<\/ol>\n<p>Six checks, under a minute, and they catch every failure mode described here. Automate them into a single script and run it from cron with output mailed only on non-zero results. If the audit reveals a host that is chronically behind because it runs out of disk during <code>apt<\/code> unpacking, the fix is capacity, not process \u2014 <a href=\"https:\/\/virtualserversvps.com\/\">sizing the instance with headroom for kernel packages and the package cache<\/a> is cheaper than untangling a half-installed dpkg state on a Friday night.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Unattended updates fail quietly. The service stays &#8220;active&#8221;, the timer fires every day, and nothing gets installed \u2014 because a package is on hold, because a repository is unreachable, or&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":6,"footnotes":""},"categories":[4],"tags":[],"class_list":["post-840","post","type-post","status-publish","format-standard","hentry","category-security-compliance"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.1 (Yoast SEO v26.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification - Virtual Servers VPS Blog<\/title>\n<meta name=\"description\" content=\"Unattended updates fail quietly. The service stays &quot;active&quot;, the timer fires every day, and nothing gets installed \u2014 because a package is on hold, because\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification\" \/>\n<meta property=\"og:description\" content=\"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification\" \/>\n<meta property=\"og:url\" content=\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/\" \/>\n<meta property=\"og:site_name\" content=\"Virtual Servers VPS Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-10T23:58:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-20T22:04:14+00:00\" \/>\n<meta name=\"author\" content=\"Virtual-Servers-Vps-Editor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Virtual-Servers-Vps-Editor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/\",\"name\":\"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification - Virtual Servers VPS Blog\",\"isPartOf\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\"},\"datePublished\":\"2026-08-10T23:58:39+00:00\",\"dateModified\":\"2026-10-20T22:04:14+00:00\",\"author\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\"},\"description\":\"Unattended updates fail quietly. The service stays \\\"active\\\", the timer fires every day, and nothing gets installed \u2014 because a package is on hold, because\",\"breadcrumb\":{\"@id\":\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/virtualserversvps.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#website\",\"url\":\"https:\/\/virtualserversvps.com\/blog\/\",\"name\":\"Virtual Servers VPS Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0\",\"name\":\"Virtual-Servers-Vps-Editor\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g\",\"caption\":\"Virtual-Servers-Vps-Editor\"},\"sameAs\":[\"https:\/\/virtualserversvps.com\/blog\"],\"url\":\"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification - Virtual Servers VPS Blog","description":"Unattended updates fail quietly. The service stays \"active\", the timer fires every day, and nothing gets installed \u2014 because a package is on hold, because","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/","og_locale":"en_US","og_type":"article","og_title":"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification","og_description":"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification","og_url":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/","og_site_name":"Virtual Servers VPS Blog","article_published_time":"2026-08-10T23:58:39+00:00","article_modified_time":"2026-10-20T22:04:14+00:00","author":"Virtual-Servers-Vps-Editor","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Virtual-Servers-Vps-Editor","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/","url":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/","name":"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification - Virtual Servers VPS Blog","isPartOf":{"@id":"https:\/\/virtualserversvps.com\/blog\/#website"},"datePublished":"2026-08-10T23:58:39+00:00","dateModified":"2026-10-20T22:04:14+00:00","author":{"@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0"},"description":"Unattended updates fail quietly. The service stays \"active\", the timer fires every day, and nothing gets installed \u2014 because a package is on hold, because","breadcrumb":{"@id":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/virtualserversvps.com\/blog\/automated-security-updates-unattended-upgrades-vps\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/virtualserversvps.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification"}]},{"@type":"WebSite","@id":"https:\/\/virtualserversvps.com\/blog\/#website","url":"https:\/\/virtualserversvps.com\/blog\/","name":"Virtual Servers VPS Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/virtualserversvps.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/82a299a8284a66ff49f97c74684724a0","name":"Virtual-Servers-Vps-Editor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/virtualserversvps.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d820b15f1cd028e97610d9adf536df7be5cb6423869967037d468d5355fa003f?s=96&d=mm&r=g","caption":"Virtual-Servers-Vps-Editor"},"sameAs":["https:\/\/virtualserversvps.com\/blog"],"url":"https:\/\/virtualserversvps.com\/blog\/author\/virtualserversvps\/"}]}},"_links":{"self":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/840","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/comments?post=840"}],"version-history":[{"count":2,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/840\/revisions"}],"predecessor-version":[{"id":1450,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/posts\/840\/revisions\/1450"}],"wp:attachment":[{"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/media?parent=840"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/categories?post=840"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/virtualserversvps.com\/blog\/wp-json\/wp\/v2\/tags?post=840"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}