Unattended Upgrades Failure Modes: Held Packages, Deferred Reboots and Patch Verification

Unattended updates fail quietly. The service stays “active”, the timer fires every day, and nothing gets installed — because a package is on hold, because a repository is unreachable, or because a previous run left the package database locked. On a single instance that you check weekly this is survivable; across a fleet it means patch latency drifts from days to months without anyone noticing. This guide covers the failure modes that matter, how to detect each one from logs and state files, and how to add the verification step that most setups are missing. It applies to any Linux instance, from a small single-purpose VPS up to a fleet of fleet-managed nodes.

How the Debian/Ubuntu Stack Fits Together

There are three moving pieces, and confusing them is the root cause of most “why didn’t my server update” incidents:

• apt-daily.timer — runs apt-get update twice a day, refreshing the package lists.
• apt-daily-upgrade.timer — runs unattended-upgrade, which installs from the refreshed lists.
• unattended-upgrades.service — the actual worker, subject to systemd sandboxing restrictions.

<code>systemctl list-timers 'apt*' --all
systemctl status unattended-upgrades.service --no-pager
cat /etc/apt/apt.conf.d/20auto-upgrades</code>

If 20auto-upgrades shows both values at 0, nothing will ever run regardless of what 50unattended-upgrades says. That is the first check, and it is the most common single-line cause of a stale fleet.

Failure Mode 1: Held Packages

apt-mark hold is sticky and easy to forget. A held package blocks that package only, but a held kernel meta-package (linux-image-generic) silently stops all kernel security updates while everything else keeps flowing — the fleet looks healthy and is missing the patches that matter most.

<code>apt-mark showhold
apt-mark showmanual | head -40
grep -c 'Packages kept back' /var/log/unattended-upgrades/unattended-upgrades.log</code>

Audit holds as part of the same routine that reviews firewall rules. Every hold should have an owner and an expiry date, or it should be removed.

Failure Mode 2: Repository and Network Failures

If apt-get update fails, unattended-upgrade does nothing and exits 0 in many configurations. That exit code is the trap: your monitoring sees success. Check for stale package lists by timestamp rather than by exit status:

<code>stat -c '%y %n' /var/lib/apt/lists/*Release | sort | head -5
date -u
grep -iE 'Err:|W:|Failed to fetch' /var/log/apt/term.log | tail -20</code>

A Release file older than 48 hours means the daily update has not completed. Common causes on a VPS: an outbound firewall rule added for a different purpose, a proxy environment variable set in one shell but not in systemd’s environment, or a mirror that no longer carries the release (EOL distributions are the classic case). Verify the proxy is visible to the service, not just to your SSH session:

<code>systemctl show apt-daily.service -p Environment
systemctl show unattended-upgrades.service -p Environment</code>

Failure Mode 3: Sandbox Restrictions Blocking the Worker

The packaged unattended-upgrades.service runs with ProtectSystem, PrivateTmp, restricted address families and a read-only view of parts of the filesystem. Hooks that write outside the permitted paths — a custom post-invoke script, a package that stages files in /var/tmp — fail with permission errors that appear only in the journal.

<code>journalctl -u unattended-upgrades --since '7 days ago' | grep -iE 'permission|read-only|denied'</code>

The correct fix is usually to remove the offending hook, not to weaken the sandbox. If a hook genuinely needs to write, move it into a separate systemd unit triggered by apt-daily-upgrade.service completing.

Failure Mode 4: Deferred Reboots Piling Up

Kernel, glibc and systemd updates do not take effect until reboot. The mechanism is /var/run/reboot-required plus the reboot-required.pkgs list:

<code>cat /var/run/reboot-required 2>/dev/null
cat /var/run/reboot-required.pkgs 2>/dev/null
uname -r
ls -1 /boot/vmlinuz-* | sort -V | tail -3</code>

If uname -r is several kernels behind the newest installed image, you are patching on disk and not in memory. This matters because the kernel is where most remotely exploitable privilege-escalation fixes land. A reboot policy is therefore part of the patch policy:

• Reboot at a defined maintenance window, not immediately after patching — unless the CVE is being actively exploited.
• Notify before rebooting and log the event with the package list that triggered it.
• If you cannot reboot, live-patch the kernel (kpatch or kernel-livepatch) — but live patching only covers the specific CVEs shipped in the patch stream, never a general upgrade.

<code># example: reboot only in a window, and only when required
cat >/usr/local/sbin/patch-reboot-check <<'EOF'
#!/bin/sh
[ -f /var/run/reboot-required ] || exit 0
logger -t patch-reboot "reboot required: $(tr '\n' ' ' < /var/run/reboot-required.pkgs)"
/sbin/shutdown -r +10 "Scheduled patching reboot"
EOF
chmod +x /usr/local/sbin/patch-reboot-check</code>

Failure Mode 5: The Locked dpkg Database

Two updaters running simultaneously — unattended-upgrades and an interactive apt upgrade, or a configuration-management run — cause the second to fail with “Could not get lock /var/lib/dpkg/lock-frontend”. Over SSH this is obvious; under a cron-based tool it becomes a silent skip.

<code>lsof /var/lib/dpkg/lock-frontend
fuser -v /var/lib/dpkg/lock 2>/dev/null</code>

Ensure exactly one updater is in charge. If configuration management also patches, disable unattended-upgrades rather than letting both race. Check with ps aux | grep -E 'apt|dpkg' at a time when a run should be in progress.

Verification: Prove Patches Actually Landed

Exit codes are not evidence. Three checks together give a real answer:

<code># 1. what the last run did
tail -30 /var/log/unattended-upgrades/unattended-upgrades.log

# 2. what is pending right now
apt-get -s upgrade | grep -c '^Inst'

# 3. what still has known CVEs
apt-get -s dist-upgrade | grep '^Inst' | grep -i security</code>

Then track the one number that matters: days since the last successful security install. A simple probe writes it into a file your monitoring already scrapes:

<code>#!/bin/sh
# write to a textfile endpoint if using node_exporter
last=$(grep 'All upgrades installed' /var/log/unattended-upgrades/unattended-upgrades.log \
       | tail -1 | awk '{print $1}')
if [ -n "$last" ]; then
  ts=$(date -d "$last" +%s 2>/dev/null)
  [ -n "$ts" ] && echo "$(( ($(date +%s) - ts) / 86400 ))" > /var/lib/node_exporter/days_since_patch.prom
fi
grep -i security /var/log/unattended-upgrades/unattended-upgrades.log | wc -l > /var/lib/node_exporter/security_pkgs_patched.prom</code>

Alert when the day count exceeds your patch SLA. This single metric catches every failure mode above, because all of them end in the same observable state: no successful security installs for an unusual length of time.

Mail and Webhook Notifications

Local mail from a VPS is usually rejected or dropped. Point it at a relay or replace it with a webhook:

<code>Unattended-Upgrade::Mail "[email protected]";
Unattended-Upgrade::MailReport "on-change";</code>

Set on-change rather than always if you route notifications to a human; always generates noise that trains people to ignore it. For automated handling, a webhook hook in /etc/apt/apt.conf.d/99webhook posting to your incident tool is more reliable than SMTP because it fails loudly.

A Minimal Weekly Audit

1. systemctl list-timers 'apt*' — confirm both timers last fired within 24 hours.
2. Check the newest Release file is under 48 hours old.
3. apt-mark showhold — empty or documented.
4. apt-get -s upgrade | grep -c '^Inst' — zero, or explainable.
5. uname -r against the installed kernel list.
6. The days-since-patch metric within SLA.

Six checks, under a minute, and they catch every failure mode described here. Automate them into a single script and run it from cron with output mailed only on non-zero results. If the audit reveals a host that is chronically behind because it runs out of disk during apt unpacking, the fix is capacity, not process — sizing the instance with headroom for kernel packages and the package cache is cheaper than untangling a half-installed dpkg state on a Friday night.