Cluster setup stuck

[Liam Young]

Deploying the cluster gets stuck with units in the following state:

mysql-innodb-cluster/0* waiting idle 8 172.20.0.4 Not all instances clustered
mysql-innodb-cluster/1 waiting idle 9 172.20.0.19 Instance not yet in the cluster
mysql-innodb-cluster/2 waiting idle 10 172.20.0.27 Instance not yet in the cluster

The lead unit has the following error in the juju unit log:
2020-05-05 06:39:34 ERROR juju-log Failed adding instance 172.20.0.19 to cluster: Logger: Tried to log to an uninitialized logger.
^[[31mERROR: ^[[0mInstance '172.20.0.19:3306' is part of the Group Replication group but is not in the metadata. Please use <Cluster>.rescan() to update the metadata.
Traceback (most recent call last):
  File "<string>", line 3, in <module>
SystemError: RuntimeError: Cluster.add_instance: Metadata inconsistent

2020-05-05 06:39:34 INFO juju-log Adding instance, 172.20.0.27, to the cluster.
2020-05-05 06:39:34 ERROR juju-log Failed adding instance 172.20.0.27 to cluster: Logger: Tried to log to an uninitialized logger.
^[[31mERROR: ^[[0mInstance '172.20.0.27:3306' is part of the Group Replication group but is not in the metadata. Please use <Cluster>.rescan() to update the metadata.
Traceback (most recent call last):
  File "<string>", line 3, in <module>
SystemError: RuntimeError: Cluster.add_instance: Metadata inconsistent

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/725529

[Liam Young (gnuoy)]

The functional tests of the following pr hit this issue. The crashdump shows the error in the logs of the mysql-innodb-cluster_0 unit

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-masakari-monitors/723613/5/5540/index.html

[OpenStack Infra (hudson-openstack)]

Change abandoned by Liam Young (<email address hidden>) on branch: master
Review: https://review.opendev.org/725529