systemd Timer OnCalendar Accuracy: Replay Storms, AccuracySec and RandomizedDelaySec on Small Hosts

On a single-vCPU or 1 GB VPS, a misconfigured OnCalendar expression combined with Persistent=true can fire dozens of jobs simultaneously the moment the host boots after downtime. systemd replays every missed activation it recorded in /var/lib/systemd/timers/, and if those timers also share an OnCalendar boundary, they collide on the same wall-clock instant. The result looks like a random load spike minutes after boot that nobody scheduled. This article shows how to observe, measure and fix that behaviour without disabling the timers you actually depend on.

See what the timer is really doing

Before editing anything, read the state systemd already keeps. Every persistent timer records its last trigger in a stamp file, and list-timers projects the next elapse for you:

systemctl list-timers --all --no-pager
systemctl status myapp-backup.timer
stat /var/lib/systemd/timers/myapp-backup.timer

The LEFT and LAST columns matter most. If LAST is stale but NEXT is seconds away after a reboot, you are about to experience a replay. The mtime of the stamp file is the exact instant systemd last honoured the timer — a useful ground truth when unit logs are noisy.

Why a replay storm happens

With Persistent=true, systemd compares the current time to the stamp file. If activations were missed, it runs the unit once immediately — not once per missed interval. That single replay is usually desirable for backups. The storm arrives when you have many timers whose schedule anchors on the same minute. On a 1 GB host, ten backups plus log maintenance jobs waking in the same second saturate both CPU and disk queue depth, and the OOM killer or I/O stall that follows is frequently misdiagnosed as a kernel bug.

Two directives control the spread:

• AccuracySec= lets systemd coalesce timers onto the same wakeup. The default is 1 minute, which is why co-scheduled jobs bunch up. Set it to a value that represents your real tolerance — 0 for precise work, larger to let the system batch wakeups and save idle power.
• RandomizedDelaySec= adds a random offset up to the given value. This is the single most effective anti-collision knob. A RandomizedDelaySec=900 spreads a fleet of hourly jobs across a fifteen-minute window.

A worked example

Consider a database dump that should run nightly but must not stampede with other maintenance. The unit that storms looks like this:

[Unit]
Description=Nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

The hardened version separates precision from collision avoidance. We keep the wall-clock anchor but give systemd permission to drift the actual start:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
AccuracySec=1s
RandomizedDelaySec=1800

Now the dump fires somewhere in the 02:00–02:30 window. Backups stay inside their retention envelope while the disk sees one write stream instead of many. After editing, reload and confirm the new schedule:

systemctl daemon-reload
systemctl restart myapp-backup.timer
systemctl list-timers myapp-backup.timer --no-pager

Verifying the spread empirically

Do not trust the config — watch the actual trigger times. The unit’s own journal records when it started, and you can extract the delta between consecutive runs to prove the randomisation is working:

journalctl -u myapp-backup.service --since "7 days ago" -o short-iso | grep -i Started

If every line shows the same 02:00:00 timestamp, RandomizedDelaySec is not taking effect — usually because the timer was reloaded without a daemon-reload, or because the drop-in is shadowed by a later unit file. Run systemctl cat myapp-backup.timer to see the merged result and confirm which file wins.

Transient timers for one-shot maintenance

For an ad-hoc job you do not want to persist to disk, systemd-run accepts the same scheduling syntax and is ideal for a deferred, randomised reboot or scrub on a production box:

systemd-run --on-calendar="*-*-* 03:00:00" --timer-property=RandomizedDelaySec=1200 \
  /usr/local/bin/fstrim-all

This creates a transient timer that disappears after it fires, so it cannot silently survive a reboot and replay weeks later.

Putting it together during a boot storm

If you are diagnosing an unexplained post-reboot spike right now, the fastest path is to correlate timer replays with resource pressure. On a host already starved for memory, the extra replay load is exactly the kind of pressure that a well-sized virtual server is meant to absorb — but sizing cannot fix a schedule that deliberately synchronises every job. Combine list-timers, the stamp-file mtimes and the unit journal to name the culprit, then apply RandomizedDelaySec and re-measure.

The principles here — observe before tuning, spread contention, verify empirically — are the same ones behind premium cloud solutions that keep maintenance windows predictable on small instances. A timer is cheap to configure; an unmanaged replay storm on a 1 GB VPS is not.