removing an instance and adding it back does not work
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
New
|
Undecided
|
Unassigned |
Bug Description
On a fresh jammy 3-unit deployment using charm revision 39 of 8.0/stable channel, trying to remove an instance and adding it back results in error:
juju run-action mysql-innodb-
the remove action fails due to bug LP#1954306 but it actually partially succeeds removing the instance:
{"address": "10.5.3.85:3306", "instanceErrors":
["NOTE: group_replication is stopped."], "memberState": "OFFLINE", "mode": "R/O",
"
The instance is not removed from the cluster, but it is taken offline and group_replication is stopped.
Trying to add it back now:
juju run-action mysql-innodb-
The action succeeds on the leader, but the status does not change. Trying to workaround this to start the group_replication back, the only action that does that is update-unit-acls, but it cannot be run due to the condition at [1]. Hacking the code to remove the condition or starting it manually, results in the following state:
{"address": "10.5.3.85:3306",
"
Fatal error: Invalid (empty) username when attempting to connect to the master
server. Connection attempt terminated. (13117) at 2023-02-09 15:42:58.656640"],
"mode": "R/O", "readReplicas": {}, "recovery": {"receiverError": "Fatal error:
Invalid (empty) username when attempting to connect to the master server. Connection
attempt terminated.", "receiverErrorN
At this point, another workaround is to forcibly removing the instance, but that hits bugs LP#2006759 and LP#1983158.
tags: | added: sts |
I'm fairly sure this is due to, or related to, "During scale-out of cluster (zaza-openstack -tests) the leader fails to join in the new instance when related to prometheus" (https:/ /bugs.launchpad .net/charm- mysql-innodb- cluster/ +bug/2015256) where the create_user() for prometheus user causes a write to the db whilst it is configured, but not yet joined, to the cluster; this causes the join_instance() to fail at that point.
I'm going to mark this as a dup, but if you feel it is not, then please un-dup it and add further comments/evidence. Thanks.