systemd Services That Die After a Reboot: journalctl and Ordering Diagnostics

A unit that starts fine when you type systemctl start and dies after a reboot has one of four causes, and each has a different fix. The efficient way to narrow it is not to read the unit file top to bottom but to query systemd for what it actually did at boot: whether the unit was pulled in, whether its dependencies were satisfied at that moment, and what exit status it produced when it was.

This page works through that sequence using only systemctl and journalctl. It assumes a systemd 252 or newer guest, which is what Debian 12, Ubuntu 24.04 and current RHEL 9 ship with. If you are sizing an instance to run these services on, our VPS plan sizing by workload covers how much RAM each service class needs beyond the unit file itself.

First: Distinguish the Four Failure Classes

Symptom after reboot	Cause	Confirming command
Unit not in systemctl list-units at all	Never enabled	systemctl is-enabled myapp → disabled
Unit listed but inactive (dead)	Started but exited 0, or Requires= dependency vanished	journalctl -b -u myapp
failed with exit code 1	Application error at startup, often missing mount or network	systemctl show -p Result,ExecMainStatus myapp
Active, then dead minutes later	Ordering race: filesystem or network not ready when the unit ran	systemd-analyze critical-chain myapp.service

Identify which row you are in before changing anything. The fix for row one (systemctl enable) is wrong and mildly harmful for row four, where enabling is already done and the problem is ordering.

Step 1: Read the Boot-Scoped Journal, Not the Current One

The most common journalctl mistake is omitting -b. Without it you get every boot interleaved, and a failure from three weeks ago can look like the cause of today’s problem. List boots first, then filter.

journalctl --list-boots | tail -5
journalctl -b -1 -u myapp --no-pager | tail -40
journalctl -b -u myapp -p warning --no-pager

-b -1 is the previous boot — the one that preceded the last reboot. -b -2 goes back further. -p warning filters to severity warning and above, which removes the informational chatter that usually makes the useful line hard to find. Add -o short-precise when you need millisecond ordering between two units.

Step 2: Ask systemd Why, Not the Application

The application’s error message is often the second-order symptom. systemd’s own record of the start attempt contains the dependency that failed and the exact result code.

systemctl show myapp.service -p Result -p ExecMainStatus -p ExecMainCode \
  -p ActiveEnterTimestamp -p InactiveEnterTimestamp
systemctl status myapp.service --no-pager -l
systemd-analyze critical-chain myapp.service

Result=exit-code with ExecMainStatus=1 means the process ran and failed. Result=resources means a limit was hit. Result=timeout — with a service that takes 40 seconds to open a database connection — means TimeoutStartSec is too short, which is common on busy hosts at boot when every service starts at once. Result=dependency means something it Requires never came up.

Step 3: Check the Three Ordering Traps

Almost all “works manually, fails at boot” reports come from one of these. Each has a specific unit-file fix.

[Unit]
Description=My application
After=network-online.target mysql.service
Wants=network-online.target
Requires=mysql.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
User=app
WorkingDirectory=/srv/app
ExecStart=/srv/app/bin/server --config /etc/app/config.yml
Restart=on-failure
RestartSec=5
# Give slow dependencies time on a loaded host
TimeoutStartSec=90

[Install]
WantedBy=multi-user.target

• Trap 1 — network.target instead of network-online.target. network.target only means the network stack was configured, not that an address exists. Services that bind to a specific IP or resolve a hostname at startup need Wants=network-online.target plus After=network-online.target.
• Trap 2 — database dependency without a health check. Requires=mysql.service is satisfied when the unit becomes active, which for MySQL is before it accepts connections. Add an ExecStartPre wait, or a drop-in with ExecStartPre=/usr/local/bin/wait-for-port 127.0.0.1 3306.
• Trap 3 — WantedBy=default.target on a system service. That places the unit in the user-session shutdown order, not the boot order, and it frequently ends up masked or skipped. Use multi-user.target.

Step 4: Test a Boot Without Rebooting

The fastest iteration loop is a private systemd instance with a modified root, so a broken unit cannot take down the SSH session you are debugging through.

sudo systemctl daemon-reload

# Validate syntax before you trust it
systemd-analyze verify /etc/systemd/system/myapp.service

# Simulate the boot ordering without rebooting
systemd-analyze critical-chain myapp.service --no-pager
systemd-analyze plot > /tmp/boot.svg

systemd-analyze verify catches misspelled directives, which systemd otherwise logs at debug level and ignores — a silent failure mode that produces exactly the “works manually” symptom. If you need a real cold-boot test on a production host, take a snapshot first; systemctl isolate rescue.target and back is faster than a full reboot but will stop every non-essential unit.

Step 5: Watch a Start in Real Time

When the failure is intermittent, attach a journal tail before you trigger the restart so you capture the whole attempt, including the units that were scheduled around it.

journalctl -f -u myapp -u mysql -u systemd-udevd -o short-precise &
sudo systemctl restart myapp
systemctl show myapp -p NRestarts -p ExecMainStartTimestampMonotonic -p ExecMainExitTimestampMonotonic

NRestarts incrementing quickly with Restart=on-failure means the unit is in a restart loop that StartLimitBurst will eventually stop. If the unit reaches failed and stays there, check systemctl show -p Result — a value of start-limit-hit tells you the loop already happened and the current status is the aftermath, not the cause.

Step 6: Make the Next Boot Prove Itself

Enabling a unit is a promise about the next reboot. Verify the promise without taking an outage by checking the install target links directly:

systemctl is-enabled myapp
ls -l /etc/systemd/system/multi-user.target.wants/ | grep myapp
systemctl list-dependencies multi-user.target --no-pager | grep -c myapp

Then add a post-boot assertion timer that reports any unit that is not active within two minutes of boot. It catches the silent cases — a unit that starts, runs for a minute, and exits — which no reboot-and-look approach reliably finds.

#!/usr/bin/env bash
# /usr/local/sbin/postboot-assert.sh
sleep 120
bad=0
for u in myapp nginx redis-server mysql; do
  systemctl is-active --quiet "$u" || { echo "DOWN: $u"; bad=1; }
done
[ $bad -eq 0 ] || journalctl -b --no-pager -u "$u" -n 30
exit $bad

Quick Reference: Journal Filters Worth Memorising

Goal	Command
Everything from this boot	journalctl -b
Previous boot only	journalctl -b -1
One unit, warnings and up	journalctl -b -u myapp -p warning
By priority, all units	journalctl -p err -b
Follow live, microsecond order	journalctl -f -o short-precise
Since a timestamp	journalctl --since "2026-10-23 14:00" --until "15:00"
Field-based (json)	journalctl -o json-pretty -u myapp
Prioritised boot summary	systemd-analyze blame / critical-chain

Two habits eliminate most of this class of incident. Enable and verify in the same session rather than weeks apart, and treat ordering directives as assertions to be tested — After= does not mean “wait for it to be ready”, only “do not start first”. For the sizing side of running several services on one small guest, our VPS plan sizing guide by workload lists RAM and CPU floors per service class.