Clock skew on testbeds

[Balint Reczey]

Clock stepped backward on a testbed:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/b/boost1.71/20200501_151103_1e387@/log.gz

...
context FAIL stderr: make[2]: Warning: File '/tmp/tmp.qPfGPM84a1/src/demo2.cpp' has modification time 0.52 s in the future
autopkgtest [13:55:37]: test context: - - - - - - - - - - stderr - - - - - - - - - -
make[2]: Warning: File '/tmp/tmp.qPfGPM84a1/src/demo2.cpp' has modification time 0.52 s in the future
make[2]: warning: Clock skew detected. Your build may be incomplete.
..

[Balint Reczey (rbalint)]

My suspect is systemd-timesyncd:

See src/timesync/timesyncd-manager.c:

/*
 * Maximum delta in seconds which the system clock is gradually adjusted
 * (slewed) to approach the network time. Deltas larger that this are set by
 * letting the system time jump. The kernel's limit for adjtime is 0.5s.
 */
#define NTP_MAX_ADJUST 0.4

I suspect the boost compilation's load made the internal clock diverge enough for systemd to jump the time.

The default PollIntervalMaxSec is 2048 seconds in systemd.

I can reduce it in systemd, but there is a jump as high as 0.85 in
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/b/boost1.71/20200317_211804_626ff@/log.gz
which could not have been prevented by just halving the default.

How about changing the autopkgtest setup to either change a default or disable systemd-timesyncd or install chrony instead?

[Balint Reczey (rbalint)]

We have a few options to fix the issue. The one I'd prefer would be switching to systemd to depend on chrony as the preferred time-daemon because timesynd turns back time and that can cause issues like this one.

An other option is disabling systemd-timesyncd in the test VMs, that would make the tests more reproducible but less similar to real installations's behavour.