Auditing Cron Job Failures and Email Alerting on a Small VPS

Cron is the most common scheduler on a VPS and the easiest to forget. A typo in a path, an expired token, a full disk — and a backup or certificate renewal silently stops running, sometimes for months, because nobody reads the mail that cron occasionally tries to send. The fix is not more cron entries; it is making every job report its outcome and architecting so that a failure is loud. This guide audits what is actually scheduled, makes jobs fail visibly, and wires reliable alerting that survives a broken local mailer.

Alerting is only as good as the box generating it, so the underlying instance must be sized to run your monitoring without contention. If jobs are crawling because the host is oversold, compare VPS plans and check the steal and disk figures before you blame your scripts — and the plan comparison helps you confirm the RAM headroom an alerting agent needs.

Step 1: Inventory Every Scheduled Job

There is more than one crontab, and the one you see with crontab -l is only your own. Enumerate them all:

for u in $(cut -d: -f1 /etc/passwd); do
  echo "== $u =="; crontab -l -u "$u" 2>/dev/null
done
ls -l /etc/cron.d /etc/cron.daily /etc/cron.hourly
systemctl list-timers --all

The last line matters because a growing number of packaged jobs run as systemd timers, not cron — if you only audit crontabs you will miss half the schedule. Record owner, command, schedule and expected runtime for each entry; a job with no owner is a job nobody will notice breaking.

Step 2: Make Cron Fail Loudly

Cron mails a job’s output to the owner only if the local MTA is configured — and on a minimal VPS it usually is not, so output is discarded. Two robust patterns avoid relying on local mail entirely.

Pattern A: wrap every job and check the exit code.

# /usr/local/bin/runjob.sh
#!/bin/bash
name="$1"; shift
start=$(date +%s)
if "$@"; then
  echo "$(date -Is) OK $name" >> /var/log/jobs.log
else
  rc=$?
  echo "$(date -Is) FAIL($rc) $name" >> /var/log/jobs.log
  /usr/local/bin/alert.sh "$name failed rc=$rc"
fi

Cron entry: 0 3 * * * /usr/local/bin/runjob.sh backup /usr/local/bin/backup.sh. Every job now writes a one-line verdict to /var/log/jobs.log, which is far easier to audit than scattered mail.

Pattern B: require explicit success. A job that prints nothing on success and logs only errors fails silently if it never runs at all. Better to emit a heartbeat: each successful run appends its timestamp, and a separate watchdog alerts if the newest heartbeat is older than 2× the schedule. That catches jobs that crash before printing anything — the exact failure the naive approach misses.

Step 3: A Watchdog for Dead Jobs

#!/bin/bash
# alert if backup heartbeat is older than 26 hours
last=$(grep 'OK backup' /var/log/jobs.log | tail -1 | awk '{print $1}')
age=$(( $(date +%s) - $(date -d "$last" +%s) ))
[ "$age" -gt 93600 ] && /usr/local/bin/alert.sh "backup heartbeat stale: $((age/3600))h"

Run this watchdog from a different scheduler context than the jobs themselves — if both live in the same crontab and cron dies, the watchdog dies with it. A systemd timer with OnCalendar=*-*-* 06:00:00 and Persistent=true is a good host, because Persistent replays a missed run after downtime.

Step 4: Alerting That Does Not Depend on Local Mail

Local sendmail frequently breaks or is blocked by providers, so send alerts straight to an HTTP endpoint where possible. A minimal alert.sh using curl to a webhook or a transactional-mail API:

#!/bin/bash
msg="[$(hostname)] $1"
curl -fsS -m 10 -X POST https://hooks.example.com/vps   -H 'Content-Type: application/json'   -d "{\"text\":\"$msg\"}" ||   logger -t alert "DELIVERY FAILED: $msg"

The || logger fallback is deliberate: if the network is down, the alert is at least captured in the journal instead of vanishing. Test it by invoking alert.sh "test" manually and confirming delivery before trusting it with real failures.

Step 5: Watch for Duplicate and Overlapping Runs

A long-running job that overlaps its next invocation can corrupt data or hammer a database. Guard with flock:

0 * * * * /usr/bin/flock -n /var/lock/mysync.lock /usr/local/bin/sync.sh

-n makes the second invocation exit immediately if the first still holds the lock, which is exactly the behaviour you want for backups and syncs.

Step 6: Log Rotation and Retention for Job Output

An append-only /var/log/jobs.log grows without bound on a busy host. Rotation keeps it useful and caps disk use, and because job verdicts are one line each, even a year of history is tiny:

# /etc/logrotate.d/jobs
/var/log/jobs.log {
    weekly
    rotate 12
    compress
    missingok
    notifempty
    create 0640 root adm
}

Keep enough history to spot a job that started failing two months ago but was never noticed. Combine the log with the heartbeat watchdog and a weekly grep for FAIL, and you have both a real-time alert and a queryable record.

Make the Audit Routine

Once a month, read the last few lines of /var/log/jobs.log, confirm every job has a recent OK, and check that systemctl list-timers shows no failed units. A quick grep turns the log into a report:

grep FAIL /var/log/jobs.log | tail -20
awk '$2 ~ /^OK/ {print $3}' /var/log/jobs.log | sort -u

Combined with heartbeat watchdogs and webhook alerting, this turns cron from a silent liability into a monitored, auditable schedule — the difference between discovering a broken backup today and discovering it when you need a restore.