Clock drift is one of the few server problems that produces symptoms nowhere near its cause. A VM whose clock has drifted a few seconds can reject TLS connections, invalidate authentication tokens, corrupt database ordering assumptions, and break distributed locks \u2014 while the CPU, memory, and disk graphs all look perfectly healthy. Virtual machines drift faster than physical hardware because the guest clock is derived from the hypervisor’s virtual timer, which is affected by scheduling and CPU migration. Here is how to detect drift and fix it properly with chrony.<\/p>\n\n\n\n
Start by measuring the offset against a known-good source:<\/p>\n\n\n\n
# Current tracking status \u2014 offset, frequency, last sync
chronyc tracking
# NTP synchronisation state of the kernel clock
timedatectl status
# Sources and their measured offsets
chronyc sources -v<\/code><\/pre>\n\n\n\nIn chronyc tracking<\/code>, the line to watch is System time<\/strong> \u2014 the current offset from real time. Sub-millisecond is healthy. Values measured in seconds mean the clock has not been disciplined properly, or chronyd<\/code> is not running at all. The Frequency<\/strong> value tells you how fast the clock is drifting in parts per million; a large frequency means chronyd<\/code> is working hard to correct a badly skewed oscillator, which is normal on VMs.<\/p>\n\n\n\nWhy VMs Drift More Than Bare Metal<\/h2>\n\n\n\n- Virtual timers<\/strong> \u2014 guest time is derived from a virtualised clock source, which is subject to host scheduling jitter.<\/li>
- vCPU migration<\/strong> \u2014 when a guest thread moves between physical cores, the TSC on the destination core may not be perfectly aligned.<\/li>
- Save\/restore and migration<\/strong> \u2014 suspended or live-migrated VMs resume with a clock that is behind by the duration of the pause.<\/li>
- Clock source choice<\/strong> \u2014
kvm-clock<\/code> is typically the most accurate source in a KVM guest; falling back to tsc<\/code> or hpet<\/code> can be measurably worse.<\/li>- Steal time<\/strong> \u2014 if the host is oversubscribed, the guest’s timer interrupts arrive late, and the clock accumulates error.<\/li><\/ul>\n\n\n\n
You can check the clock source and confirm the hypervisor exposes a paravirtualised clock:<\/p>\n\n\n\n
cat \/sys\/devices\/system\/clocksource\/clocksource0\/current_clocksource
cat \/sys\/devices\/system\/clocksource\/clocksource0\/available_clocksource
dmesg | grep -i -E 'kvm-clock|clocksource'<\/code><\/pre>\n\n\n\nInstall and Configure chrony<\/h2>\n\n\n\n
chrony is the default on RHEL-family and modern Debian\/Ubuntu images, and it outperforms the older ntpd<\/code> in virtualised environments because it converges faster after a VM resumes from a pause. A reasonable base configuration:<\/p>\n\n\n\n# \/etc\/chrony\/chrony.conf (Debian\/Ubuntu)
pool 2.debian.pool.ntp.org iburst maxsources 4
# Step the clock if the offset is large \u2014 useful after VM restore
makestep 1.0 3
# Record drift so it converges faster on restart
driftfile \/var\/lib\/chrony\/chrony.drift
# Do not serve time to the network unless required
port 0
# Log statistics for post-incident review
logdir \/var\/log\/chrony
log measurements statistics tracking<\/code><\/pre>\n\n\n\nThe makestep 1.0 3<\/code> directive is important on VMs. It instructs chrony to step the clock \u2014 rather than slowly slew it \u2014 if the offset exceeds one second during the first three updates after startup. This is exactly the case that occurs after a VM snapshot resume, where the clock can be minutes behind.<\/p>\n\n\n\nApply and verify:<\/p>\n\n\n\n
sudo systemctl restart chrony
chronyc tracking
chronyc sources -v
timedatectl status<\/code><\/pre>\n\n\n\nWhat Breaks When the Clock Is Wrong<\/h2>\n\n\n\nSystem<\/th> Symptom of drift<\/th> Tolerance<\/th><\/tr><\/thead> TLS certificates<\/td> "Certificate not yet valid" \/ expired errors<\/td> Seconds to minutes<\/td><\/tr> JWT \/ OAuth tokens<\/td> Token rejected as expired or not-yet-valid<\/td> Typically 30\u2013300 s<\/td><\/tr> Kerberos<\/td> Authentication failures (replay protection)<\/td> ~5 minutes<\/td><\/tr> Database replication<\/td> Stale reads, inconsistent timestamps, GTID issues<\/td> Milliseconds to seconds<\/td><\/tr> Distributed locks<\/td> Split-brain, duplicate job execution<\/td> Milliseconds<\/td><\/tr> Log correlation<\/td> Events appear out of order across hosts<\/td> Seconds<\/td><\/tr> Cron<\/td> Jobs skipped or run twice<\/td> Seconds<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe database row deserves emphasis. Multi-node replication, time-based partitioning, and audit logging all assume clocks are monotonic and comparable. In a multi-server deployment, even a two-second skew between primary and replica can produce confusing replication errors that look like application bugs.<\/p>\n\n\n\n
Monitoring Clock Offset<\/h2>\n\n\n\n
Treat clock offset as a monitored metric, not a one-time fix. Export it to your monitoring system so a drift that exceeds threshold pages you before applications fail:<\/p>\n\n\n\n
# Raw offset in seconds, suitable for a textfile collector
chronyc tracking | awk '\/System time\/ {print $4}'
# One-line health check for cron
chronyc tracking | awk '\/System time\/ {v=$4; if (v<0) v=-v; if (v>0.5) print \"DRIFT ALERT: \" v \"s\"}'<\/code><\/pre>\n\n\n\nA useful threshold is 100 ms for alerting and 1 s for paging. If you are running databases or authentication services on the same host, tighten it.<\/p>\n\n\n\n
Post-Restore and Post-Migration Handling<\/h2>\n\n\n\n