Clock drift is one of the few server problems that produces symptoms nowhere near its cause. A VM whose clock has drifted a few seconds can reject TLS connections, invalidate authentication tokens, corrupt database ordering assumptions, and break distributed locks — while the CPU, memory, and disk graphs all look perfectly healthy. Virtual machines drift faster than physical hardware because the guest clock is derived from the hypervisor’s virtual timer, which is affected by scheduling and CPU migration. Here is how to detect drift and fix it properly with chrony.
Confirm You Have a Drift Problem
Start by measuring the offset against a known-good source:
# Current tracking status — offset, frequency, last sync
chronyc tracking
# NTP synchronisation state of the kernel clock
timedatectl status
# Sources and their measured offsets
chronyc sources -v
In chronyc tracking, the line to watch is System time — the current offset from real time. Sub-millisecond is healthy. Values measured in seconds mean the clock has not been disciplined properly, or chronyd is not running at all. The Frequency value tells you how fast the clock is drifting in parts per million; a large frequency means chronyd is working hard to correct a badly skewed oscillator, which is normal on VMs.
Why VMs Drift More Than Bare Metal
- Virtual timers — guest time is derived from a virtualised clock source, which is subject to host scheduling jitter.
- vCPU migration — when a guest thread moves between physical cores, the TSC on the destination core may not be perfectly aligned.
- Save/restore and migration — suspended or live-migrated VMs resume with a clock that is behind by the duration of the pause.
- Clock source choice —
kvm-clockis typically the most accurate source in a KVM guest; falling back totscorhpetcan be measurably worse. - Steal time — if the host is oversubscribed, the guest’s timer interrupts arrive late, and the clock accumulates error.
You can check the clock source and confirm the hypervisor exposes a paravirtualised clock:
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource
dmesg | grep -i -E 'kvm-clock|clocksource'
Install and Configure chrony
chrony is the default on RHEL-family and modern Debian/Ubuntu images, and it outperforms the older ntpd in virtualised environments because it converges faster after a VM resumes from a pause. A reasonable base configuration:
# /etc/chrony/chrony.conf (Debian/Ubuntu)
pool 2.debian.pool.ntp.org iburst maxsources 4
# Step the clock if the offset is large — useful after VM restore
makestep 1.0 3
# Record drift so it converges faster on restart
driftfile /var/lib/chrony/chrony.drift
# Do not serve time to the network unless required
port 0
# Log statistics for post-incident review
logdir /var/log/chrony
log measurements statistics tracking
The makestep 1.0 3 directive is important on VMs. It instructs chrony to step the clock — rather than slowly slew it — if the offset exceeds one second during the first three updates after startup. This is exactly the case that occurs after a VM snapshot resume, where the clock can be minutes behind.
Apply and verify:
sudo systemctl restart chrony
chronyc tracking
chronyc sources -v
timedatectl status
What Breaks When the Clock Is Wrong
| System | Symptom of drift | Tolerance |
|---|---|---|
| TLS certificates | "Certificate not yet valid" / expired errors | Seconds to minutes |
| JWT / OAuth tokens | Token rejected as expired or not-yet-valid | Typically 30–300 s |
| Kerberos | Authentication failures (replay protection) | ~5 minutes |
| Database replication | Stale reads, inconsistent timestamps, GTID issues | Milliseconds to seconds |
| Distributed locks | Split-brain, duplicate job execution | Milliseconds |
| Log correlation | Events appear out of order across hosts | Seconds |
| Cron | Jobs skipped or run twice | Seconds |
The database row deserves emphasis. Multi-node replication, time-based partitioning, and audit logging all assume clocks are monotonic and comparable. In a multi-server deployment, even a two-second skew between primary and replica can produce confusing replication errors that look like application bugs.
Monitoring Clock Offset
Treat clock offset as a monitored metric, not a one-time fix. Export it to your monitoring system so a drift that exceeds threshold pages you before applications fail:
# Raw offset in seconds, suitable for a textfile collector
chronyc tracking | awk '/System time/ {print $4}'
# One-line health check for cron
chronyc tracking | awk '/System time/ {v=$4; if (v<0) v=-v; if (v>0.5) print "DRIFT ALERT: " v "s"}'
A useful threshold is 100 ms for alerting and 1 s for paging. If you are running databases or authentication services on the same host, tighten it.
Post-Restore and Post-Migration Handling
Every time a VM is restored from a snapshot or live-migrated, assume its clock is wrong. If you run automated snapshot-based backups, add a clock check to the restore verification step. A server that comes back with a five-minute-old clock will silently fail TLS handshakes until the offset is corrected — and if makestep is not configured, chrony will slew the correction over hours instead of fixing it immediately.
On a full-virtualisation instance with its own kernel, you control the clock source, the chrony configuration, and the step policy directly. Rental plans at virtualserversvps.com run full VMs rather than shared containers, so kernel-level time settings are yours to tune rather than fixed by the host’s container policy.
Checklist
- Verify
chronydis running andtimedatectlreports active synchronisation. - Confirm the clock source is
kvm-clock(orxen) rather than a slow fallback. - Set
makestep 1.0 3so post-restore clock jumps are corrected instantly. - Configure a drift file so convergence is fast after reboot.
- Monitor offset continuously and alert at 100 ms.
- Re-check the clock after every snapshot restore and live migration.
- Align all hosts in a cluster to the same upstream sources, so skew between them stays bounded.

Leave a Reply
You must be logged in to post a comment.