In bionic, one of the ceph packages installed causes chrony to auto-install even on lxd

[Drew Freiberger]

When running ceph-mon in containers, best practice is to have chrony/ntp configured and installed on the bare metal and then have the container trust the system clock, as the container should not manage the system clock.

It seems that in source=cloud:bionic-stein, ceph-base package ends up installing chrony automatically as a recommended package.

chrony should not be installed in the LXD context of this charm. If it's possible to remove chrony from the recommended package list or to have the charm remove chrony if ceph-base is installed in a lxd, that wouuld be preferable to having to adding the NTP charm to configure chrony in a lxd, or to have to disable the chrony service.

[Drew Freiberger (afreiberger)]

Perhaps just masking the service would be sufficient if running in a lxd.

[Alex Kavanagh (ajkavanagh)]

TRIAGE: Confirmed, but not sure whether to fix in packages or "make the problem go away" in the charm; more discussion needed.

[Felipe Reyes (freyes)]

I think the charm is the one responsible of disabling chrony, something like this would do the trick.

if is_container():
    subprocess.check_call("sudo timedatectl set-ntp off")

[Bryan Quigley (bryanquigley)]

"sudo timedatectl set-ntp off" didn't work for me disabling chrony service

sudo systemctl disable chrony was needed.

[Bryan Quigley (bryanquigley)]

I would suggest the charm should just do the apt install with --no-install-recommends

That doesn't install chrony..

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gábor Mészáros (gabor.meszaros)]

I've just found this piece of code change, that removes the removal of ntp when we're running in container:
https://review.opendev.org/#/c/584051/5/lib/ceph/utils.py@a43

originating from here: NTP implementation hard-coded ;https://bugs.launchpad.net/charm-ceph-mon/+bug/1780690

Is there a way to revert back the
if is_container():
   PACKAGES.remove('ntp')

code snippet?

[Ponnuvel Palaniyappan (pponnuvel)]

Notes:

1. `--no-install-recommends` also excludes `ceph-common` package from getting installed which is needed (it could possibly exclude more packages, too).

2. `ntp` isn't getting installed - it's not an issues/related any more.

So simply removing `chrony` is probably enough.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738647

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ponnuvel Palaniyappan (<email address hidden>) on branch: master
Review: https://review.opendev.org/738647

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738664

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738763

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738781

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ponnuvel Palaniyappan (<email address hidden>) on branch: master
Review: https://review.opendev.org/738763

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ponnuvel Palaniyappan (<email address hidden>) on branch: master
Review: https://review.opendev.org/738781

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738664
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=60a9a4f27aa1cddd717431abae7a26c46f5f8bd5
Submitter: Zuul
Branch: master

commit 60a9a4f27aa1cddd717431abae7a26c46f5f8bd5
Author: Ponnuvel Palaniyappan <email address hidden>
Date: Tue Jun 30 17:27:09 2020 +0100

    Remove chrony if inside a container

    When running ceph-mon in containers, best practice is
    to have chrony/ntp configured and installed on the bare
    metal and then have the container trust the system
    clock, as the container should not manage the system
    clock.

    The chrony package get installed automatically as
    part of the dependencies of other packages, which
    gets removed in this change.

    Also contains related changes for charms.ceph.

    Change-Id: If8beb28ea5b5e6317180e52c3e32463e472276f4
    Closes-Bug: #1852441
    Depends-On: Ie3c9c5899c1d46edd21c32868938d3290db321e7

[Ponnuvel Palaniyappan (pponnuvel)]

Quoting Ante Karamtic:
"If ntp/chrony is removed from ceph-mon, then ntp charm goes into error state if it's installed on ceph-mon units.

On other machines, ntp charm detects that it's in the container and then reports that it is a container and that there's nothing to do.

In case of ceph-mon, now it goes into error state because chrony is not there. So ntp charm should update the status before it checks if chrony is installed."

[Ponnuvel Palaniyappan (pponnuvel)]

I am not sure if there's anything to fix here for the ntp charm.

The ntp charm shouldn't really be installed in a container.
In general, It should be installed alongside a principal charm
that's on a bare-metal machine.

In situations like charm upgrade, The ntp charm could end up installing
the chrony package again even if it's a container where it was previously
removed. And ceph-mon charm could remove it again.

In the case of ntp being a sub-ordinate of ceph-mon container
and when ceph-mon removes chrony package, the ntp unit goes into
'blocked' state which seems reasonable to me given it
shouldn't be there in the container in the firt place.

So the only things I could think of are:
 - update ntp charm's doc that it shouldn't be installed in a container
 - provide a clear error message if ntp is deployed on a container

OTOH, if there's a legitimate use-case to deploy ntp in a container,
then I don't see to be done here.

Thoughts?