Intermittent deployment failure, unit stuck in "Instance not yet in the cluster" state
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
In Progress
|
High
|
David Ames |
Bug Description
$ juju status mysql-innodb-
Model Controller Cloud/Region Version SLA Timestamp
zaza-4928e0d8aa32 fnordahl-
App Version Status Scale Charm Store Rev OS Notes
mysql-innodb-
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-
mysql-innodb-
mysql-innodb-
Machine State DNS Inst id Series AZ Message
0 started 10.5.0.4 48dfc189-
1 started 10.5.0.33 2c7d7ef7-
2 started 10.5.0.32 59223555-
Excerpt from leader unit log:
2020-06-02 09:58:01 ERROR juju-log cluster:2: Failed adding instance 10.5.0.33 to cluster: Logger: Tried to log to an uninitialized logger.
WARNING: A GTID set check of the MySQL instance at '10.5.0.33:3306' determined that it contains transactions that do not originate from the cluster, which must be discarded before it can join the cluster.
10.5.0.33:3306 has the following errant GTIDs that do not exist in the cluster:
cb5028c2-
WARNING: Discarding these extra GTID events can either be done manually or by completely overwriting the state of 10.5.0.33:3306 with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.
Having extra GTID events is not expected, and it is recommended to investigate this further and ensure that the data can be removed prior to choosing the clone recovery method.
Clone based recovery selected through the recoveryMethod option
NOTE: Group Replication will communicate with other members using '10.5.0.33:33061'. Use the localAddress option to override.
Validating instance configuration at 10.5.0.33:3306...
This instance reports its own address as 10.5.0.33:3306
Instance configuration is suitable.
A new instance will be added to the InnoDB cluster. Depending on the amount of
data on the cluster this might take from a few seconds to several hours.
Adding instance to the cluster...
ERROR: Unable to start Group Replication for instance '10.5.0.33:3306'. Please check the MySQL server error log for more information.
Traceback (most recent call last):
File "<string>", line 3, in <module>
SystemError: RuntimeError: Cluster.
Changed in charm-mysql-innodb-cluster: | |
status: | New → Triaged |
importance: | Undecided → High |
assignee: | nobody → David Ames (thedac) |
milestone: | none → 20.10 |
Changed in charm-mysql-innodb-cluster: | |
milestone: | 20.10 → 21.01 |
Changed in charm-mysql-innodb-cluster: | |
milestone: | 21.01 → none |
Changed in charm-mysql-innodb-cluster: | |
milestone: | none → 21.04 |
Hit this again, new crashdump for reference