manila-share service restarted when certificates-relation-changed(upgrading vault)

[Seyeong Kim]

the customer has issue with manila-ganesha service after they upgraded vault charm.

after upgrading, manila-ganesha charm get certificates-relation-changed hook and it is trying to stop manilna-share.
then install nfs-ganesha-ceph.

WARNING unit.manila-ganesha/5.certificates-relation-changed logger.go:60 Synchronizing state of manila-share.service with SysV service script with /lib/systemd/systemd-sysv-install.
WARNING unit.manila-ganesha/5.certificates-relation-changed logger.go:60 Executing: /lib/systemd/systemd-sysv-install disable manila-share
WARNING unit.manila-ganesha/5.certificates-relation-changed logger.go:60 Created symlink /etc/systemd/system/manila-share.service → /dev/null.
INFO unit.manila-ganesha/5.juju-log server.go:316 certificates:383: Installing ['nfs-ganesha-ceph'] with options: ['--option=Dpkg::Options::=--force-confold']

and the customer faces downtime.

I guess certificates-relation-changed hook should not restart manila-share service?

I'm analyzing my local env as well. and it calls the same hook when vault is upgraded.

I'll update the case if there is anything more.

[Jorge Merlino (jorge-merlino)]

I'll add more context to this bug after some investigation. Actually manila-ganesha is masking the
manila-share service every time the certificates-relation-changed hook is called. This occurs through the charms.openstack code that manages this hook that eventually ends up calling the install() method of manila ganesha which intentionally masks manila-share with this reasoning:

# Use goal-state to determine if we are expecting multiple units
# and, if so, mask the manila-share service so that it only ever
# gets run by pacemaker.

The backtrace to get to the install() method call is this:

File "/var/lib/juju/agents/unit-manila-ganesha-0/charm/hooks/certificates-relation-changed", line 22, in <module>
    main()
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
File "/var/lib/juju/agents/unit-manila-ganesha-0/charm/reactive/layer_openstack.py", line 150, in default_configure_certificates
    instance.configure_tls(tls)
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/classes.py", line 987, in configure_tls
    self.configure_apache()
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/classes.py", line 774, in configure_apache
    self.install()

I think it is important to understand if manila-ganesha actually needs a relation with vault and if it does not we should not allow it to be established.

[Alex Kavanagh (ajkavanagh)]

So a little more digging in the charm indicates that the payload needs keystone auth credentials and thus, in a TLS environment, does need a relation to vault in order to get the CA chain so that it can TLS auth the connection to a https endpoint for keystone.

What's wrong is that it shouldn't be trying to set up apache or anything like that, as it doesn't have an API to configure (i.e. it's a service that uses other services, but doesn't extend an API to other services to use; that's done in the manila charm).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/922087

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/923383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/923383
Committed: https://opendev.org/openstack/charm-manila-ganesha/commit/ddba2d84315eb147d8f5be623878f3ee6bf98176
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit ddba2d84315eb147d8f5be623878f3ee6bf98176
Author: Alex Kavanagh <email address hidden>
Date: Mon Jun 17 12:12:36 2024 +0100

    Use client cert for keystone-identity

    When manila-ganesha is related to vault it needs a client cert to
    configure the keystone-auth section of manila.conf to communicate with
    keystone. This patch sets that up and removes the broken server cert
    auto configuration which ended up masking the manila-share service.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1244
    Change-Id: I55e9aa09b88684517d4052dc56eed0cab05a0262
    Closes-Bug: #2064487

[Alex Kavanagh (ajkavanagh)]

Currently, the intention is only to backport this fix to the yoga version of the charm as that tends to be the version that really only started using the manila + ceph-fs solution in production.

[Alex Kavanagh (ajkavanagh)]

Also note that this ceph-fs charm bug [1] affects manila-ganesha services, so checking that it is fixed is necessary prior to deploying a manila-ganesha + ceph-fs solution.

[1] https://bugs.launchpad.net/charm-ceph-fs/+bug/2071780

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/925659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/925660

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/925661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/925662