systemd-sysv-install enable neutron-ovn-metadata-agent is called every update-status hook

[Xav Paice]

Every time update-status runs, the command systemd-sysv-install enable neutron-ovn-metadata-agent is called, causing a systemctl daemon-reload.

This shouldn't be running so often, only on config changes.

Logs:

Mar 15 20:45:33 compute-server-1 systemd[1]: Reloading.
Mar 15 20:45:33 compute-server-1 systemd[1]: message repeated 2 times: [ Reloading.]
Mar 15 20:45:33 compute-server-1 systemd[1]: Reloading.
Mar 15 20:45:34 compute-server-1 systemd[1]: Reloading.
Mar 15 20:45:34 compute-server-1 systemd[1]: Reloading.
Mar 15 20:45:34 compute-server-1 systemd[1]: Reloading.
```

Hunted the unit logs, found this in unit-ovn-chassis-0.log:

```
2021-03-15 20:45:31 INFO juju-log Creating bridge br-int
2021-03-15 20:45:31 INFO juju-log Creating bridge br-data
2021-03-15 20:45:31 INFO juju-log Adding port bond0 to bridge br-data
2021-03-15 20:45:32 INFO juju-log Invoking reactive handler: reactive/ovn_chassis_charm_handlers.py:83:configure_ovs
2021-03-15 20:45:32 INFO juju-log CompletedProcess(args=('ovs-vsctl', '--no-wait', 'set-ssl', '/etc/ovn/key_host', '/etc/ovn/cert_host', '/etc/ovn/ovn-chassis.crt'), returncode=0, stdout='')
2021-03-15 20:45:32 INFO juju-log CompletedProcess(args=('ovs-vsctl', 'set', 'open', '.', 'external-ids:ovn-encap-type=geneve', '--', 'set', 'open', '.', 'external-ids:ovn-encap-ip=10.35.82.13', '--', 'set', 'open', '.', 'external-ids:system-id=compute-server-1.fqdn.dummy'), returncode=0, stdout='')
2021-03-15 20:45:32 INFO juju-log CompletedProcess(args=('ovs-vsctl', 'set', 'open', '.', 'external-ids:ovn-remote=ssl:someip:6642,ssl:someip:6642,ssl:someip:6642'), returncode=0, stdout='')
2021-03-15 20:45:32 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2021-03-15 20:45:32 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2021-03-15 20:45:32 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:77:check_really_is_update_status
2021-03-15 20:45:32 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:88:run_default_update_status
2021-03-15 20:45:33 DEBUG juju-log tracer>
tracer: cleared flag run-default-update-status
tracer: -- dequeue handler reactive/layer_openstack.py:88:run_default_update_status
2021-03-15 20:45:33 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:121:default_request_certificates
2021-03-15 20:45:33 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:163:default_config_rendered
2021-03-15 20:45:33 DEBUG update-status active
2021-03-15 20:45:33 DEBUG update-status Synchronizing state of neutron-ovn-metadata-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
2021-03-15 20:45:33 DEBUG update-status Executing: /lib/systemd/systemd-sysv-install enable neutron-ovn-metadata-agent
2021-03-15 20:45:34 DEBUG update-status active
2021-03-15 20:45:34 INFO juju-log Invoking reactive handler: hooks/relations/ovsdb-subordinate/provides.py:104:broken:ovsdb-subordinate
2021-03-15 20:45:34 INFO juju-log Invoking reactive handler: hooks/relations/juju-info/requires.py:24:broken:juju-info
2021-03-15 20:45:34 INFO juju-log Invoking reactive handler: hooks/relations/ovsdb/requires.py:34:joined:ovsdb
2021-03-15 20:45:34 INFO juju-log ovsdb: OVSDBRequires -> joined
2021-03-15 20:45:35 INFO juju-log ovsdb: OVSDBRequires -> joined
2021-03-15 20:45:35 INFO juju-log Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates
2021-03-15 20:45:35 DEBUG juju-log Running _assess_status()
2021-03-15 20:45:35 DEBUG update-status active

[Xav Paice (xavpaice)]

Added https://github.com/openstack/charm-layer-openstack/commit/49b1075506d7625d04a05e2cb232d20ea82f6406 which might resolve this, would be good to get confirmation.

Charm version on site: cs:ovn-chassis-4, bionic/ussuri, Juju 2.7.8

[Alex Kavanagh (ajkavanagh)]

It's not a charms.openstack bug as all handlers are invoked from charm layers, so I expect the bug is in one of the ovn layers.

[Alex Kavanagh (ajkavanagh)]

I suspect the actual issue is in "configure_ovs()" in charm-layer-ovn/reactive/ovn_chassis_charm_handlers.py

If this is gated using "is-update-status-hook" then it wouldn't run during update-status (as Xav indicates).

However, this might require a bit of testing; charms.reactive charms have an (annoying) habit of 'repairing' themselves during update-status hooks where any left-over things they should've done during the last actual relations/config/upgrade/whatever hook can get done. By doing the gating, we may inadvertently break the charm.

[Alex Kavanagh (ajkavanagh)]

Although the issue is in the charm-layer-ovn, the charm still needs to be rebuilt to pick up any changes.

[Frode Nordahl (fnordahl)]

Does this still occur with most recent version of the charm?

It does appear to already be fixed with https://github.com/openstack-charmers/charm-layer-ovn/commit/6eb72f74baedd5e102cb974602bd0efa41ca5452

[Alex Kavanagh (ajkavanagh)]

Woah, I didn't notice that those fixes were in. Please dis-regard my statement. Those changes, based on the date, should be in the 21.01 released charms as Frode indicated; Xav, please could you see if the charm has those fixes?

[Xav Paice (xavpaice)]

I've requested a window to upgrade the charms in that particular environment, hoping to get back to you.

[Launchpad Janitor (janitor)]

[Expired for charm-ovn-chassis because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for charm-layer-ovn because there has been no activity for 60 days.]