mysql is stuck at executing database-peers-relation-changed hook

[Nobuto Murata]

After running the resize operation from 1 to 3 nodes in the multi-node scenario in:
https://microstack.run/docs/multi-node

It doesn't complete the resize operation in time and a mysql unit is stuck for more than two hours of executing the database-peers-relation-changed hook.

$ time sunbeam cluster resize
Warning: Permanently added '10.0.123.11' (ED25519) to the list of known hosts.
⠇ Deploying OpenStack Control Plane to Kubernetes (this may take a while) ... waiting for services to come online (73/73)Timed out while waiting for model 'openstack' to be ready
Error: Timed out while waiting for model 'openstack' to be ready

real 90m22.877s
user 5m56.278s
sys 0m21.449s

$ juju status -m openstack mysql
Model Controller Cloud/Region Version SLA Timestamp
openstack sunbeam-controller sunbeam-microk8s/localhost 3.4.2 unsupported 09:13:38Z

SAAS Status Store URL
microceph active local admin/controller.microceph

App Version Status Scale Charm Channel Rev Address Exposed Message
mysql 8.0.35-0ubuntu0.22.04.1 active 3 mysql-k8s 8.0/stable 127 10.152.183.128 no

Unit Workload Agent Address Ports Message
mysql/0* active executing 10.1.32.206 Primary
mysql/1 active idle 10.1.193.213
mysql/2 active idle 10.1.186.21

$ juju show-status-log -m openstack mysql/0 --days 1

...

20 May 2024 06:58:02Z juju-unit executing running database-relation-joined hook for horizon-mysql-router/2
20 May 2024 06:58:07Z juju-unit executing running database-relation-joined hook for placement-mysql-router/2
20 May 2024 06:58:13Z juju-unit executing running database-relation-changed hook for neutron-mysql-router/2
20 May 2024 06:58:18Z juju-unit executing running database-relation-changed hook for placement-mysql-router/2
20 May 2024 06:58:22Z juju-unit executing running database-relation-changed hook for horizon-mysql-router/2
20 May 2024 06:58:26Z juju-unit idle
20 May 2024 06:59:02Z juju-unit executing running commands
20 May 2024 06:59:15Z juju-unit idle
20 May 2024 07:00:01Z juju-unit executing running commands
20 May 2024 07:00:09Z juju-unit idle
20 May 2024 07:01:00Z juju-unit executing running commands
20 May 2024 07:01:10Z juju-unit idle
20 May 2024 07:02:01Z juju-unit executing running commands
20 May 2024 07:02:10Z juju-unit idle
20 May 2024 07:03:01Z juju-unit executing running commands
20 May 2024 07:03:21Z workload active Primary
20 May 2024 07:03:32Z juju-unit executing running database-peers-relation-changed hook

So the hook is onging from 07:03:32Z to 09:13:38Z at least.

[Nobuto Murata (nobuto)]

> So the hook is onging from 07:03:32Z to 09:13:38Z at least.

unit-mysql-0: 07:03:34 DEBUG unit.mysql/0.juju-log database-peers:11: ops 2.10.0 up and running.
unit-mysql-0: 07:03:34 DEBUG unit.mysql/0.juju-log database-peers:11: Emitting Juju event database_peers_relation_changed.
unit-mysql-0: 07:03:41 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels
unit-mysql-0: 07:03:47 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels
unit-mysql-0: 07:03:52 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels
unit-mysql-0: 07:03:57 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels
unit-mysql-0: 07:04:01 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels
unit-mysql-0: 07:04:06 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels
unit-mysql-0: 07:04:11 DEBUG unit.mysql/0.juju-log database-peers:11: Updating pod labels

-> no output after this timestamp.