[2023/edge] *-mysql-router hook failed: "update-status"

[Matt Verran]

Seemingly random drops into this state when enabling features. Cinder, Placement and Glance seem to be more affected than others but unsure if this is relevant. Switching to channel 8.0/edge throws 'Upgrade incompatible. Rollback to previous revision with `juju refresh`'

A sample trace from juju debug-log (in this case cinder-ceph)...

unit-cinder-ceph-mysql-router-0: 09:31:49 ERROR unit.cinder-ceph-mysql-router/0.juju-log Failed to bootstrap router
logged_command=['--bootstrap', 'relation-93:***@cinder-mysql-primary.openstack.svc.cluster.local:3306', '--strict', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications']
stderr:
Error: It appears that a router instance named 'system' has been previously configured in this host. If that instance no longer exists, use the --force option to overwrite it.

unit-cinder-ceph-mysql-router-0: 09:31:49 ERROR unit.cinder-ceph-mysql-router/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/./src/kubernetes_charm.py", line 177, in <module>
    ops.main.main(KubernetesRouterCharm)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/venv/ops/framework.py", line 342, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/venv/ops/framework.py", line 928, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/src/abstract_charm.py", line 216, in reconcile
    workload_.enable(tls=self._tls_certificate_saved, unit_name=self.unit.name)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/src/workload.py", line 216, in enable
    self._bootstrap_router(tls=tls)
  File "/var/lib/juju/agents/unit-cinder-ceph-mysql-router-0/charm/src/workload.py", line 188, in _bootstrap_router
    raise Exception("Failed to bootstrap router") from None
Exception: Failed to bootstrap router

[Matt Verran (mv-2112)]

$ snap list openstack
Name Version Rev Tracking Publisher Notes
openstack 2023.2 329 2023.2/edge canonical✓ -

[David Torrey (dotj)]

Also seeing this on 2023.1 multi-node setup on one node out of three.

Name Version Rev Tracking Publisher Notes
openstack 2023.1 324 2023.1/stable canonical✓ -

Wondering if this fix is relevant: https://bugs.launchpad.net/charm-mysql-router/+bug/1919560