ubuntu_ltp_syscalls failed to run on Focal s390x, unable to restart systemd-timesyncd.service

[Po-Hsu Lin]

Issue found on 5.4.0-34.38 s390x LPAR / zVM

We need to stop timesyncd for the following tests:
['leapsec01','stime01','settimeofday01','clock_settime01']

But it looks like when we're trying to restart it:
   systemctl start systemd-timesyncd

It will fail with:
  Failed to start systemd-timesyncd.service: Unit systemd-timesyncd.service is masked.

This issue did not happen on 5.4.0-33.37, the ubuntu_ltp_syscalls test can finish with that kernel.

[Po-Hsu Lin (cypressyew)]

Looks like this has something to do with systemd,
with zVM boot with 5.4.0-33, I can still reproduce this issue:

ubuntu@kernel04:~$ sudo systemctl stop systemd-timesyncd
ubuntu@kernel04:~$ sudo systemctl start systemd-timesyncd
Failed to start systemd-timesyncd.service: Unit systemd-timesyncd.service is masked.
ubuntu@kernel04:~$

[Dimitri John Ledkov (xnox)]

If it is masked, is it running?

If it is not running, why are you trying to stop it?

[Po-Hsu Lin (cypressyew)]

Hello Dimitri,

After a quick check it looks like this service is still in inactive (dead) state even after a reboot.

ubuntu@kernel04:/lib/systemd$ sudo systemctl status systemd-timesyncd
● systemd-timesyncd.service
     Loaded: masked (Reason: Unit systemd-timesyncd.service is masked.)
     Active: inactive (dead)
ubuntu@kernel04:/lib/systemd$ file /lib/systemd/system/systemd-timesyncd.service
/lib/systemd/system/systemd-timesyncd.service: cannot open `/lib/systemd/system/systemd-timesyncd.service' (No such file or directory)

And there is no "systemd-timesyncd.service" file at where is should be.

So yeah it's not running, our testing tool did not check the state first before trying to stop / start it.

$ sudo systemctl edit systemd-timesyncd
Cannot edit systemd-timesyncd.service: unit is masked.

Let me see if I can un-mask it.

[Po-Hsu Lin (cypressyew)]

The correct file location for systemd-timesyncd.service should be:

$ sudo systemctl enable systemd-timesyncd
Failed to enable unit: Unit file /etc/systemd/system/systemd-timesyncd.service is masked.
$ file /etc/systemd/system/systemd-timesyncd.service
/etc/systemd/system/systemd-timesyncd.service: symbolic link to /dev/null

[Po-Hsu Lin (cypressyew)]

I tried to remove the /etc/systemd/system/systemd-timesyncd.service symbolic link to /dev/null by following this post [1]

After that it's complaining the service was not found:
$ sudo systemctl start systemd-timesyncd
Failed to start systemd-timesyncd.service: Unit systemd-timesyncd.service not found.

Hmm...

[1] https://blog.ruanbekker.com/blog/2017/12/09/unmask-a-masked-service-in-systemd/

[Po-Hsu Lin (cypressyew)]

Re-install systemd-timesyncd seem to fix this issue.

However while trying to install systemd-timesyncd the ntp package will be removed.
Maybe we're using ntp instead of systemd-timesyncd on Focal s390x by default?

[Dimitri John Ledkov (xnox)]

systemd-timesyncd package is the default...

Please provide a tarball of /var/log such that we could trace if / when / why ntp got installed.

Or please try to grep for ntp recursively in /var/log as root to see if you can see if something is manually installing ntp (and/or if install or upgrade process was broken).

[Po-Hsu Lin (cypressyew)]

This issue can be found on Focal Azure 5.4

The attachment is from the a deployed F-azure-5.4 instance
$ dpkg -l | grep ntp
$ dpkg -l | grep timesyncd
rc systemd-timesyncd 245.4-4ubuntu3.1 amd64 minimalistic service to synchronize local time with NTP servers

A quick recursive grep for timesyncd shows it's been removed here:
Start-Date: 2020-06-03 22:37:42
Commandline: apt-get -y install walinuxagent cifs-utils linux-azure keyutils chrony
Install: libwbclient0:amd64 (2:4.11.6+dfsg-0ubuntu1.2, automatic), cifs-utils:amd64 (2:6.9-1), libtalloc2:amd64 (2.3.0-3ubuntu1, automatic), chrony:amd64 (3.5-6ubuntu6.1), keyutils:amd64 (1.6-6ubuntu1), libtevent0:amd64 (0.10.1-4, automatic), walinuxagent:amd64 (2.2.46-0ubuntu1)
Remove: systemd-timesyncd:amd64 (245.4-4ubuntu3.1)
End-Date: 2020-06-03 22:37:50

[Po-Hsu Lin (cypressyew)]

BTW nothing special for ntp recursive grep

[Balint Reczey (rbalint)]

So the time-daemon in use is chrony. Please restart chrony if it tests the kernel of install systemd-timesyncd if you would like to test systemd-timesyncd.

[Po-Hsu Lin (cypressyew)]

Modify the test, only try to stop systemd-timesyncd when it's running.
https://kernel.ubuntu.com/git/ubuntu/autotest-client-tests.git/commit/?id=d64ab9921ce1b43bee676251ff1b57562b99e85b