MAAS servers have two NTP clients

[Vern Hart]

Ubuntu comes with systemd-timesyncd to server as an NTP client. Apparently this has been the case since 16.04.

When you snap install MAAS, you end up with two NTP clients. systemd-timesyncd and chrony (which also act as a server).

This could potentially lead to problems with time synchronization. If MAAS (chrony) is configured with a local time server that is out of sync with ntp.ubuntu.com, you could experience dueling NTP clients. At the very least, you will see time sync requests heading to ntp.ubuntu.com despite MAAS being configured to use different NTP servers.

It seems to me that MAAS should be disabling the existing NTP client when installing a new one but I'm not sure how it would go about doing that. I suppose this would require apparmor rules to allow the snap to communicate with systemctl. Or maybe we just need to call this out in the documentation.

As a work-around, I've added the following to my deployment guide:

  sudo systemctl disable systemd-timesyncd
  sudo systemctl stop systemd-timesyncd

I've experience this on MAAS 3.2.6 but I expect this is on ALL MAAS versions that use chrony.

[Nobuto Murata (nobuto)]

MAAS snap is confined not classic so it doesn't have ability to adjust the system(host) clock out of the box if I'm not mistaken.

$ snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.2.6-12016-g.19812b4da 23947 3.2/stable canonical✓ -

Then, the snap is connected to "time-control" slot. And at least the doc says "the system date and time to be changed (...) via systemd-timedated".
https://snapcraft.io/docs/time-control-interface

$ snap connections maas
Interface Plug Slot Notes
avahi-observe maas:avahi-observe :avahi-observe -
content - maas:maas-logs -
content[maas-cli] maas:maas-cli maas-cli:maas-cli -
content maas:test-db-socket - -
hardware-observe maas:hardware-observe :hardware-observe -
home maas:home :home -
kernel-module-observe maas:kernel-module-observe :kernel-module-observe -
mount-observe maas:mount-observe :mount-observe -
network maas:network :network -
network-bind maas:network-bind :network-bind -
network-control maas:network-control :network-control -
network-observe maas:network-observe :network-observe -
snap-refresh-control maas:snap-refresh-control :snap-refresh-control -
system-observe maas:system-observe :system-observe -
time-control maas:time-control :time-control -

[Björn Tillenius (bjornt)]

We're currently dealing with this for a backlog feature item to redesign the region-rack model.

[Jerzy Husakowski (jhusakowski)]

This behaviour of MAAS needs to be called out in the docs.

[Bill Wear (billwear)]

I have a few questions, specifically, if we disable systemd-timesyncd, will any of the following considerations be issues?

Other local services and applications: Could there be services and applications running on system that depend on accurate time synchronization, but have specific requirements or dependencies on systemd-timesyncd? Disabling it could potentially affect their functionality.

Network time dependencies: Are any other devices or systems in the network that rely on time synchronization explicitly from systemd-timesyncd? Disabling it may impact the time synchronization of those devices.

Compatibility and configuration: Can the NTP server provided by MAAS can handle the workload and maintain accurate time across the network for all potential clients?

[Vern Hart (vern)]

These are all very interesting questions. We disabled this at our customer deployment back in January and there have been no reported issues. That doesn't necessarily mean that there are no issues, just that the customer has not noticed them (though I believe they would have).

It may be important to note that systemd-timesyncd is client-only. This means that there will be no devices or systems outside of the server that would expect to communicate with systemd-timesyncd. This also means that the workload performance of chrony vs systemd-timesyncd is not a valid comparison since chrony is a server (and client) whereas systemd-timesyncd does not serve ntp.

[Bill Wear (billwear)]

fixed; the following text has been added to all installation-relevant parts of the documentation:

When installing MAAS on Ubuntu, there can be conflicts between the existing NTP client, systemd-timesyncd, and the NTP client/server provided by MAAS, chrony. This can lead to time synchronization issues, especially if MAAS is configured with different upstream NTP servers than the ones used by systemd-timesyncd. To avoid conflicts, users can manually disable and stop systemd-timesyncd using the following command:

sudo systemctl disable --now systemd-timesyncd