manila-share service restarted when certificates-relation-changed(upgrading vault)

[Seyeong Kim]

the customer has issue with manila-ganesha service after they upgraded vault charm.

after upgrading, manila-ganesha charm get certificates-relation-changed hook and it is trying to stop manilna-share.
then install nfs-ganesha-ceph.

WARNING unit.manila-ganesha/5.certificates-relation-changed logger.go:60 Synchronizing state of manila-share.service with SysV service script with /lib/systemd/systemd-sysv-install.
WARNING unit.manila-ganesha/5.certificates-relation-changed logger.go:60 Executing: /lib/systemd/systemd-sysv-install disable manila-share
WARNING unit.manila-ganesha/5.certificates-relation-changed logger.go:60 Created symlink /etc/systemd/system/manila-share.service → /dev/null.
INFO unit.manila-ganesha/5.juju-log server.go:316 certificates:383: Installing ['nfs-ganesha-ceph'] with options: ['--option=Dpkg::Options::=--force-confold']

and the customer faces downtime.

I guess certificates-relation-changed hook should not restart manila-share service?

I'm analyzing my local env as well. and it calls the same hook when vault is upgraded.

I'll update the case if there is anything more.

[Jorge Merlino (jorge-merlino)]

I'll add more context to this bug after some investigation. Actually manila-ganesha is masking the
manila-share service every time the certificates-relation-changed hook is called. This occurs through the charms.openstack code that manages this hook that eventually ends up calling the install() method of manila ganesha which intentionally masks manila-share with this reasoning:

# Use goal-state to determine if we are expecting multiple units
# and, if so, mask the manila-share service so that it only ever
# gets run by pacemaker.

The backtrace to get to the install() method call is this:

File "/var/lib/juju/agents/unit-manila-ganesha-0/charm/hooks/certificates-relation-changed", line 22, in <module>
    main()
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
File "/var/lib/juju/agents/unit-manila-ganesha-0/charm/reactive/layer_openstack.py", line 150, in default_configure_certificates
    instance.configure_tls(tls)
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/classes.py", line 987, in configure_tls
    self.configure_apache()
File "/var/lib/juju/agents/unit-manila-ganesha-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/classes.py", line 774, in configure_apache
    self.install()

I think it is important to understand if manila-ganesha actually needs a relation with vault and if it does not we should not allow it to be established.

[Alex Kavanagh (ajkavanagh)]

So a little more digging in the charm indicates that the payload needs keystone auth credentials and thus, in a TLS environment, does need a relation to vault in order to get the CA chain so that it can TLS auth the connection to a https endpoint for keystone.

What's wrong is that it shouldn't be trying to set up apache or anything like that, as it doesn't have an API to configure (i.e. it's a service that uses other services, but doesn't extend an API to other services to use; that's done in the manila charm).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/922087

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/charm-manila-ganesha/+/923383