openstack: networking-ovn-metadata-agent configuration not re-rendered on charm upgrade

[Frode Nordahl]

The configuration for ``networking-ovn-metadata-agent`` is currently not re-rendered on charm upgrade.

This is due to the reconfiguration of OVS being guarded behind certificate changes, probably as a side effect of the calls ``configure_ovs`` make to OVS may hang when already configured and that the method restarts daemons at the end.

We should make the ``configure_ovs`` method compare run-time configuration prior to making calls to OVS and make the call to restart daemons happen only when strictly necessary.

[Edward Hope-Morley (hopem)]

We have had reports of this causing significant performance degradation in environments using SRIOV so perhaps time to look at getting it resolved.

[Edward Hope-Morley (hopem)]

Here is an example:

update-status hook every 5m:

# grep "Synchronizing state of neutron-ovn-metadata-agent.service" /var/log/juju/unit-ovn-chassis-0.log| tail -n 5
2024-03-19 15:16:14 WARNING unit.ovn-chassis/0.update-status logger.go:60 Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
2024-03-19 15:21:59 WARNING unit.ovn-chassis/0.update-status logger.go:60 Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
2024-03-19 15:26:46 WARNING unit.ovn-chassis/0.update-status logger.go:60 Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
2024-03-19 15:31:01 WARNING unit.ovn-chassis/0.update-status logger.go:60 Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
2024-03-19 15:35:56 WARNING unit.ovn-chassis/0.update-status logger.go:60 Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.

# grep systemd /var/log/syslog | grep "Reloading"| tail -n 5
Mar 19 15:35:55 juju-2dd04f-sf00380684-sncf-12 systemd[1]: Reloading.
Mar 19 15:35:55 juju-2dd04f-sf00380684-sncf-12 systemd[1]: Reloading.
Mar 19 15:35:56 juju-2dd04f-sf00380684-sncf-12 systemd[1]: Reloading.
Mar 19 15:35:56 juju-2dd04f-sf00380684-sncf-12 systemd[1]: Reloading.
Mar 19 15:35:56 juju-2dd04f-sf00380684-sncf-12 systemd[1]: Reloading.

[Edward Hope-Morley (hopem)]

So actually it isn't configure_ovs(). The charm will run [1] on every update-status hook which in turn calls [2] and then [3]. This results in a call to service_resume() for all services (in this case neutron-ovn-metadata-agent and ovn-host) at [4] which translates to a systemctl umask then enable which produces the stderr msg we see in the unit log i.e.

# systemctl enable neutron-ovn-metadata-agent
Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable neutron-ovn-metadata-agent

[1] https://github.com/openstack/charm-layer-openstack/blob/7c671b0696977f455616565d956895b2f890464b/reactive/layer_openstack.py#L173
[2] https://github.com/openstack/charms.openstack/blob/aec72e735b4678e3c8970dbfffe16b002a05b812/charms_openstack/charm/classes.py#L258
[3] https://github.com/juju/charm-helpers/blob/511bc0346e3002cb31475c03697c2f5b065967f3/charmhelpers/contrib/openstack/utils.py#L1605
[4] https://github.com/juju/charm-helpers/blob/511bc0346e3002cb31475c03697c2f5b065967f3/charmhelpers/core/host.py#L242

[Edward Hope-Morley (hopem)]

This appears to be default behaviour for any charm that uses layer_openstack since nothing ever unsets the flags that are causing that method to be called each time a hook fires. It can be overriden by the ovn-chassis charm and I feel like perhaps it should be since there is no need to be continuously doing this for a service that is already enabled.

[Edward Hope-Morley (hopem)]

opened https://bugs.launchpad.net/charm-ovn-chassis/+bug/2058505 since my issue seems to be slightly different to the one this bug was originally opened for.