Inconsistent relation data when leadership changes as only leader sets relation data - causes mayhem with related mysql-router units (was: new added units try to join a new cluster)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
Triaged
|
High
|
Unassigned | ||
MySQL Router Charm |
Invalid
|
Undecided
|
Unassigned | ||
vault-charm |
Invalid
|
Undecided
|
Unassigned |
Bug Description
bundle: openstack-2023.2
vault charm: 1.8/stable rev 209
ubuntu: 22.04
I have a vault 3 node cluster but after restarting my etcd containers every new vault units I add, they join a new cluster instead of joining the existing one.
What is happened: I had 3 vault units ready and clustered, I restarted the etcd nodes and 2 vault units became off the cluster, so I removed them and then added another 2 units but they never joined the vault cluster.
The new added units keep remaining in "Vault needs to be initialized" while I'd rather expect "unit is sealed".
I tryed also to remove relations with etcd and then recreating, but it didn't work.
looking into the logs of a new added unit it seems to be that it try to join a new cluster with the other new ones even they are all sealed:
unit-vault-15: 13:51:44 INFO unit.vault/
unit-vault-15: 13:51:45 INFO juju.worker.
unit-vault-15: 13:51:46 INFO unit.vault/
unit-vault-15: 13:51:47 ERROR unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
unit-vault-15: 13:51:48 INFO unit.vault/
these are the status of the 3 unit, the first is the older with the existing cluster and the other 2 are the new ones:
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.8.8
Build Date n/a
Storage Type mysql
Cluster Name vault-cluster-
Cluster ID 7fa4c481-
HA Enabled true
HA Cluster https:/
HA Mode active
Active Since 2024-04-
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.8.8
Build Date n/a
Storage Type raft
HA Enabled true
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.8.8
Build Date n/a
Storage Type raft
HA Enabled true
I tryied also to unseal the new units but it failed because they are not initialized.
The strange is also that storage type is raft and not mysql like the first unit.
So, How can add new units to an existing cluster?
description: | updated |
description: | updated |
description: | updated |
summary: |
- new added units try to join a new cluster + Inconsistent relation data when leadership changes as only leader sets + relation data - causes mayhem with related mysql-router units (was: new + added units try to join a new cluster) |
Changed in charm-mysql-innodb-cluster: | |
importance: | Undecided → High |
Changed in charm-mysql-router: | |
status: | Triaged → Invalid |
Hi @macchese
Please could you add a juju_status.txt of the model, and the logs from the vault units. Also "juju show-unit" for each of the vault units would be very helpful. This might explain why the vault units are not clustering.
Thanks.