When deployed on FCE infra node with maas snap, two chrony daemons run

[Drew Freiberger]

Background:

When BootStack manages an environment, we add a maas-infra model to the juju controller connected to MAAS and add the infra machines hosting MAAS and related pod VMs as manual provider machines to the model and then deploy LMA charms for monitoring of the infrastructure nodes and services.

The process for this is to run the following:

juju add-model maas-infra
juju add-machine ssh:ubuntu@<infra ip> # resultant machine should be 0

juju deploy ubuntu --to 0
juju deploy ntp
juju deploy nrpe
juju add-relation ntp ubuntu
juju add-relation ntp:nrpe-external-master nrpe:nrpe-external-master
juju add-relation nrpe:monitors <nagios-remote-offer-of-monitors from the openstack or kubernetes model>

In the past, with deb packaged MAAS installations, this would have the ntp charm taking over the package configuration of the /etc/chrony configurations. However, with snapped MAAS, the MAAS team has added chronyd as a service within the snap and is configured with a snappy-path to the chrony config.

In this snapped maas instance, we then have two chrony services running, with no idea which service is listening on the ntp port at any given moment (after reboots, service cycles, etc), no consistency in who's configuring the ntp service that is running, and potentially two chrony services referencing upstream clocks and trying to discipline the same clock.

Resultant process list looks like:
_chrony 827784 1 0 Jul29 ? 00:00:09 /usr/sbin/chronyd
root 2018415 1699229 0 Jul20 ? 00:00:19 /snap/maas/12555/usr/sbin/chronyd -u root -d -f /var/snap/maas/12555/etc/chrony/chrony.conf

It would be super handy for the NTP charm to detect MAAS installations and run in "monitoring only" mode so that MAAS maintains control over the NTP service configuration and the ntpmon nrpe checks still function as they would if the NTP charm had configured the service.

To note, the chronyc command to monitor the snapped maas installation is "/snap/maas/current/usr/bin/chronyc". Both the ntpmon nagios plugin and the charm will require adjustment to support this configuration so that chrony packages don't get installed in the root OS if MAAS has installed chronyc via snap.

[Drew Freiberger (afreiberger)]

I've also filed https://github.com/paulgear/ntpmon/issues/13 for the ntpmon fixup required.

[Drew Freiberger (afreiberger)]

When charm ntp and maas snap are co-resident, two chronyd services start and listen. The winner of the race at boot time determines which clients can connect and which configuration is used to manage the local clock.

The ntp charm is typically the one we want to win the configuration battle, so charm-ntp should detect if ntp/chrony is installed via maas snap and then configure the maas chrony config file /var/snap/maas/current/etc/chrony/chrony.conf and manage the chrony service via maas snap.

The issue is not just in monitoring, but also accessibility/reachability for peers.