Unattended-Upgrades Failure Modes on a VPS: Holding Kernels, Deferred Reboots, and the Dry-Run That Lies

Automatic patching is normally described as install-and-forget. The reality on a long-lived VPS is a set of specific failure modes: a held package that blocks every subsequent security update, a kernel that is installed but never booted, an apt-daily timer that collides with your own maintenance window, and a --dry-run that reports success while doing nothing. Each is verifiable from the command line in under a minute.

Confirm what is actually being applied

The Origins-Pattern list in /etc/apt/apt.conf.d/50unattended-upgrades decides everything. If ${distro_id}:${distro_codename}-security is present but -updates is not, you are patching CVEs and skipping bugfixes that sometimes carry security-relevant backports. More importantly, the release codename in that file is written from the file in /etc/os-release at install time; after an in-place release upgrade the pattern can silently reference a codename that no longer has any sources.

apt-config dump APT::Periodic::Unattended-Upgrade
ls -l /var/lib/apt/periodic/
journalctl -u unattended-upgrades --since '14 days ago' | grep -Ei 'no .*package|not upgrading|held'
/usr/lib/apt/apt.systemd.daily --dry-run 2>&1 | tail -40

The --dry-run integration test writes a lock file into /var/lib/apt/periodic/ and can leave a stale one behind if interrupted. A stale unattended-upgrades-stamp combined with a stale download-upgradeable-stamp is the most common reason a working configuration stops running: the timer starts, sees a lock, and exits 0 without logging anything alarming. Delete stale stamps only after confirming no apt process holds the dpkg lock.

Held packages block the whole transaction

apt resolves upgrades transactionally. One package forced to hold – typically a hand-patched nginx or a pinned PHP – causes all packages depending on it to be skipped, which often includes the security update you actually care about. Enumerate the blocking set before it bites:

apt-mark showhold
APT_LISTCHANGES_FRONTEND=none apt-get -s dist-upgrade \
  | grep -E 'kept back|not upgraded|held'
# read the dependency chain for one held package
apt-get -s install --only-upgrade nginx 2>&1 | head -20

Symptom in journal	Root cause	Verification
No log entries for days	stale stamp or missing timer	systemctl list-timers apt-daily*
‘kept back’ lines repeat	package hold or phased update	apt-mark showhold, apt-get -s dist-upgrade
Downloaded but not applied	APT::Periodic::Install-OnShutdown off	apt-config dump | grep Install-OnShutdown
Applies then reverts	image/config management re-pins version	diff your CM state against dpkg versions

Phased updates make patching look late

Ubuntu’s phased-update system withholds a package from a percentage of machines for days to catch regressions. This is by design, but it means unattended-upgrades legitimately reports nothing to do while a security fix is available upstream. If you need the fix immediately, override per-package rather than disabling phasing globally:

apt-get -s upgrade | grep -i phased
# force a specific package past phasing
cat >/etc/apt/apt.conf.d/99-phasing-override <<'EOF'
APT::Get::Always-Include-Phased-Updates "true";
EOF
apt-get -o APT::Get::Always-Include-Phased-Updates=true install --only-upgrade openssl

Leaving the global override in place defeats the safety mechanism; scope it to the single invocation, as shown. On a fleet where every node shares a template, phasing is often the only thing preventing a simultaneous fleet-wide regression, so keep it enabled by default. For a broader look at hardening decisions that interact with patching - SSH exposure, auditd rules, and firewall posture - see the security-oriented work summarized alongside our VPS hosting specifications and isolation model.

Kernels install; they do not boot

linux-image-generic upgrades land in /boot and sit idle until a reboot. The classic consequence is a host reporting a patched kernel version while uname -r still shows the vulnerable one. Detect the divergence explicitly:

uname -r
ls -1 /boot/vmlinuz-* | sed 's|/boot/vmlinuz-||' | sort -V | tail -3
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required
# what reboot reason
apt-get -s autoremove 2>/dev/null | grep -i 'linux-image'
dpkg -l 'linux-image-*' | awk '/^ii/{print $2, $3}'

The needrestart tool answers the adjacent question - which running services still hold pre-patch libraries in memory. In non-interactive mode it returns a status you can alert on, which is more reliable than parsing reboot-required alone:

NEEDRESTART_MODE=l needrestart -b 2>/dev/null | grep -E 'SERVICES|KERNEL'
# exit code 1 = services need restart, 2 = kernel reboot needed

Map those exit codes to an alert and pair it with an explicit maintenance policy: reboot weekly during a defined window, allow up to one kernel behind on a spare node, and never let the gap exceed the vendor's supported patch interval. Prune old kernels with apt autoremove --purge only after confirming the running kernel is not the one being removed - on small instances /boot filling up is itself a patching outage, and a full /boot will fail every subsequent kernel upgrade while security updates to userspace keep succeeding, masking the problem until the next reboot.

Making the reboot decision auditable

"Reboot weekly" is a policy, not a mechanism. Make it observable so that a missed window is a signal rather than a surprise. Record the running kernel and the newest installed kernel in a text file each boot, then diff against the previous entry. A growing number of lines means the host is accumulating pending kernels and the maintenance window is not being honoured.

cat >/etc/cron.daily/kernel-drift <<'EOF'
#!/bin/sh
NEW=$(ls -1 /boot/vmlinuz-* | sed 's|/boot/vmlinuz-||' | sort -V | tail -1)
RUN=$(uname -r)
LOG=/var/log/kernel-drift.log
[ "$NEW" != "$RUN" ] && echo "$(date -Is) pending=$NEW running=$RUN" >> "$LOG"
sed -i '1,200!d' "$LOG"
EOF
chmod +x /etc/cron.daily/kernel-drift
# sanity check for /boot capacity before the next kernel lands
df -h /boot
apt-get -s --purge autoremove | grep -c '^Remv linux-image'

Run the same drill against a rebuilt instance periodically. A patch pipeline that has never been exercised end to end is an assumption, not a control, and instance-level snapshots let you validate it without risking the production host - the capabilities and restore behaviour are described on our VPS hosting features overview.