Cluster stuck with status: Not all instances configured for clustering
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
Triaged
|
High
|
Unassigned |
Bug Description
In test run https:/
================
mysql-innodb-
filebeat/68 active idle 10.246.65.16 Filebeat ready.
landscape-
logrotated/63 active idle 10.246.65.16 Unit is ready.
nrpe/78 active idle 10.246.65.16 icmp,5666/tcp Ready
prometheus-
telegraf/68 active idle 10.246.65.16 9103/tcp Monitoring mysql-innodb-
mysql-innodb-
filebeat/20 active idle 10.246.64.226 Filebeat ready.
landscape-
logrotated/15 active idle 10.246.64.226 Unit is ready.
nrpe/26 active idle 10.246.64.226 icmp,5666/tcp Ready
prometheus-
telegraf/20 active idle 10.246.64.226 9103/tcp Monitoring mysql-innodb-
mysql-innodb-
filebeat/72 active idle 10.246.65.22 Filebeat ready.
landscape-
logrotated/66 active idle 10.246.65.22 Unit is ready.
nrpe/80 active idle 10.246.65.22 icmp,5666/tcp Ready
prometheus-
telegraf/72 active idle 10.246.65.22 9103/tcp Monitoring mysql-innodb-
================
In the logs of the leader unit we can see that it fails to connect to the other units:
================
2023-04-08 13:41:55 ERROR unit.mysql-
Traceback (most recent call last):
File "<string>", line 1, in <module>
mysqlsh.DBError: MySQL Error (2003): Dba.configure_
2023-04-08 13:41:55 DEBUG unit.mysql-
2023-04-08 13:41:56 INFO unit.mysql-
2023-04-08 13:41:58 ERROR unit.mysql-
ERROR: Unable to connect to the target instance '192.168.
Traceback (most recent call last):
File "<string>", line 3, in <module>
mysqlsh.DBError: MySQL Error (2003): Cluster.
2023-04-08 13:41:58 INFO unit.mysql-
2023-04-08 13:42:01 ERROR unit.mysql-
Traceback (most recent call last):
File "<string>", line 1, in <module>
mysqlsh.DBError: MySQL Error (2003): Dba.configure_
2023-04-08 13:42:01 DEBUG unit.mysql-
2023-04-08 13:42:02 INFO unit.mysql-
2023-04-08 13:42:04 ERROR unit.mysql-
ERROR: Unable to connect to the target instance '192.168.
Traceback (most recent call last):
File "<string>", line 3, in <module>
mysqlsh.DBError: MySQL Error (2003): Cluster.
================
In the non-leader units there are no errors and it does indicate that it found peers:
================
2023-04-08 13:30:51 INFO unit.mysql-
2023-04-08 13:30:51 DEBUG unit.mysql-
2023-04-08 13:30:52 DEBUG unit.mysql-
================
I'm not sure what the cause of the connection issue is.
Crashdumps and configs can be found here:
https:/
tags: | added: cdo-qa foundations-engine |
Changed in charm-mysql-innodb-cluster: | |
status: | Incomplete → Triaged |
importance: | Undecided → High |
So the 'reason' it didn't cluster is due to the leader not being able to contact the followers:
023-04-08 13:41:49 INFO unit.mysql- innodb- cluster/ 1.juju- log server.go:316 Invoking reactive handler: reactive/ mysql_innodb_ cluster_ handlers. py:138: check_quorum innodb- cluster/ 1.juju- log server.go:316 Found peers: 192.168. 33.214, 192.168. 33.219 innodb- cluster/ 1.juju- log server.go:316 Expect 2 peers innodb- cluster/ 1.juju- log server.go:316 Reached quorum innodb- cluster/ 1.juju- log server.go:316 Invoking reactive handler: reactive/ mysql_innodb_ cluster_ handlers. py:172: configure_ instances_ for_clustering innodb- cluster/ 1.juju- log server.go:316 Configuring instances for clustering. innodb- cluster/ 1.juju- log server.go:316 Configuring instance for clustering: 192.168.33.214. innodb- cluster/ 1.juju- log server.go:316 Failed configuring instance 192.168.33.214: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory instance: Can't connect to MySQL server on '192.168.33.214' (113)
2023-04-08 13:41:49 DEBUG unit.mysql-
2023-04-08 13:41:49 DEBUG unit.mysql-
2023-04-08 13:41:49 DEBUG unit.mysql-
2023-04-08 13:41:49 INFO unit.mysql-
2023-04-08 13:41:49 DEBUG unit.mysql-
2023-04-08 13:41:49 INFO unit.mysql-
2023-04-08 13:41:55 ERROR unit.mysql-
Traceback (most recent call last):
File "<string>", line 1, in <module>
mysqlsh.DBError: MySQL Error (2003): Dba.configure_
2023-04-08 13:41:55 DEBUG unit.mysql- innodb- cluster/ 1.juju- log server.go:316 Checking cluster status. innodb- cluster/ 1.juju- log server.go:316 Adding instance, 192.168.33.214, to the cluster. innodb- cluster/ 1.juju- log server.go:316 Failed adding instance 192.168.33.214 to cluster: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory 33.214: 3306'. Please verify the connection settings, make sure the instance is available and try again. add_instance: Could not open connection to '192.168. 33.214: 3306': Can't connect to MySQL server on '192.168.33.214' (113)
2023-04-08 13:41:56 INFO unit.mysql-
2023-04-08 13:41:58 ERROR unit.mysql-
ESC[31mERROR: ESC[0mUnable to connect to the target instance '192.168.
Traceback (most recent call last):
File "<string>", line 3, in <module>
mysqlsh.DBError: MySQL Error (2003): Cluster.
2023-04-08 13:41:58 INFO unit.mysql- innodb- cluster/ 1.juju- log server.go:316 Configuring instance for clustering: 192.168.33.219. innodb- cluster/ 1.juju- log server.go:316 Failed configuring instance 192.168.33.219: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory instance: Can't connect to MySQL server on '192.168.33.219' (113)
2023-04-08 13:42:01 ERROR unit.mysql-
Traceback (most recent call last):
File "<string>", line 1, in <module>
mysqlsh.DBError: MySQL Error (2003): Dba.configure_
2023-04-08 13:42:01 DEBUG unit.mysql- innodb- cluster/ 1.juju- log server.go:316 Checking cluster status. innodb- cluster/ 1.juju- log server.go:316 Adding instance, 192.168.33.219, to the cluster. innodb- cluster/ 1.juju- log server.go:316 Failed adding instance 192.168.33.219 to cluster: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory
2023-04-08 13:42:02 INFO unit.mysql-
2023-04-08 13:42:04 ERROR unit.mysql-
ESC[31mERROR: ESC[0mUnable to connect to the target instance '1...