mysql cluster stuck in waiting while adding instances, already replicating

[Alexander Balderson]

deploying mysql-cluster on aws, the 3 mysql instances are stuck in waiting, saying they arnt clusterd.

The leader unit shows that its trying to cluster, but failing:

2021-09-29 07:16:50 ERROR juju-log Failed adding instance 172.31.42.38 to cluster: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory

Clone based recovery selected through the recoveryMethod option

Validating instance configuration at 172.31.42.38:3306...
This instance reports its own address as ^[[1m172.31.42.38:3306^[[0m
Instance configuration is suitable.
^[[36mNOTE: ^[[0mGroup Replication will communicate with other members using '172.31.42.38:33061'. Use the localAddress option to override.

A new instance will be added to the InnoDB cluster. Depending on the amount of
data on the cluster this might take from a few seconds to several hours.

Adding instance to the cluster...

Traceback (most recent call last):
  File "<string>", line 3, in <module>
mysqlsh.DBError: MySQL Error (3190): Cluster.add_instance: RESET MASTER is not allowed because Group Replication is running.

If the replication has actually started, then the units should report "executing," but there also seems to be some issue with connection between the instances.

Crashdump for this run can be found:
https://oil-jenkins.canonical.com/artifacts/cb5f77a0-26c6-4c30-aa84-df321161c89c/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-09-29-07.24.49.tar.gz

instances of this bug can be found here:
https://solutions.qa.canonical.com/bugs/bugs/bug/1945505

[Konstantinos Kaskavelis (kaskavel)]

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)

[Bas de Bruijne (basdbruijne)]

Opening the bug back up because we hit it again in:

https://solutions.qa.canonical.com/v2/testruns/1425c38e-a178-4c9f-b964-46e3763d5069

Crashdumps and configs are here:
https://oil-jenkins.canonical.com/artifacts/1425c38e-a178-4c9f-b964-46e3763d5069/index.html

[Alex Kavanagh (ajkavanagh)]

I'm triaging this to low as it's not clear how often this is happening; this was on the 8.0/stable 39 focal charm (current is 43). There's been work in this area in the charm, so this bug may be fixed.