Encountered here: https://review.opendev.org/#/c/743364/
A new unit has been added to the cluster after a removal of a different unit. Juju status:
mysql-innodb-cluster/1 active executing 2 172.17.110.3 Unit is ready: Mode: R/W
mysql-innodb-cluster/2* active executing 3 172.17.110.33 Unit is ready: Mode: R/O
mysql-innodb-cluster/4 blocked idle 6 172.17.110.30 MySQL InnoDB Cluster not healthy: None
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-mysql-innodb-cluster/743364/1/6528/index.html
tracer: -- dequeue handler reactive/mysql_innodb_cluster_handlers.py:329:configure_certificates
2020-07-28 18:55:25 DEBUG juju-log certificates:4: Running _assess_status()
2020-07-28 18:55:26 DEBUG certificates-relation-changed active
2020-07-28 18:55:27 DEBUG juju-log certificates:4: Opening db connection for root@localhost
2020-07-28 18:55:27 DEBUG juju-log certificates:4: Checking cluster status.
2020-07-28 18:55:53 ERROR juju-log certificates:4: Cluster is unavailable: Logger: Tried to log to an uninitialized logger.
Traceback (most recent call last):
File "<string>", line 2, in <module>
SystemError: RuntimeError: Dba.get_cluster: This function is not available through a session to an instance belonging to an unmanaged replication group
There isn't a lot of information about this error (0 docs) but I managed to find some in the mysql-shell repo:
https://github.com/mysql/mysql-shell/blob/8.0.21/modules/adminapi/common/preconditions.cc#L858-L862
void check_preconditions(const std::string &function_name,
const Cluster_check_info &state,
FunctionAvailability *custom_func_avail) {
// ...
case GRInstanceType::GroupReplication:
error +=
" to an instance belonging to an unmanaged replication "
"group";
break;
Unit tests also suggest that getCluster is not available for standalone instances:
https://github.com/mysql/mysql-shell/blob/8.0.21/unittest/scripts/auto/js_adminapi/validation/dba_preconditions.js#L31-L33
//@# Dba_preconditions_unmanaged_gr, get_cluster_fails
// getCluster is not allowed on standalone instances
||Dba.getCluster: This function is not available through a session to an instance belonging to an unmanaged replication group
Based on the error logs from multiple units, I can see that the 4th unit left the group at some point which triggered the error since it was no longer a part of the group at that point:
mysql-innodb-cluster/4
2020-07-28T17:57:11.305946Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.21-0ubuntu0.20.04.3' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
2020-07-28T17:57:11.401793Z 10 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-07-28T17:57:14.710418Z 0 [Warning] [MY-013426] [Repl] Plugin group_replication reported: 'Member version is read compatible with the group.'
2020-07-28T17:57:14.710600Z 2 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 172.17.110.3:3306.'
2020-07-28T17:57:15.711402Z 0 [ERROR] [MY-013467] [Repl] Plugin group_replication reported: 'No valid or ONLINE members exist to get the missing data from the group. For cloning check if donors of the same version and with clone plugin installed exist. For incremental recovery check if you have donors where the required data was not purged from the binary logs.'
2020-07-28T17:57:15.711510Z 0 [ERROR] [MY-011712] [Repl] Plugin group_replication reported: 'The server was automatically set into read only mode after an error was detected.'
2020-07-28T17:57:16.712503Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 172.17.110.30:3306, 172.17.110.3:3306, 172.17.110.33:3306 on view 15959464471091338:11.'
2020-07-28T17:57:20.913218Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
mysql-innodb-cluster/2 unit (R/O) also encountered many invalid replication timestamps - this is likely due to clock skews in an overcommitted infrastructure environment:
2020-07-28T16:24:13.300814Z 840 [Warning] [MY-010956] [Server] Invalid replication timestamps: original commit timestamp is more recent than the immediate commit timestamp. This may be an issue if delayed replication is active. Make sure that servers have their clocks set to the correct time. No further message will be emitted until after timestamps become valid again.
2020-07-28T16:24:13.571277Z 840 [Warning] [MY-010957] [Server] The replication timestamps have returned to normal values.
Either way, we need to handle status checks better in that regard as this does not help a lot:
mysql-innodb-cluster/4 blocked idle 6 172.17.110.30 MySQL InnoDB Cluster not healthy: None
This seems to be still an issue. Hybrid512 is seeing this consistently on fresh deployments, both with rev. 3 (20.10) and rev. 5 (21.01) of charm-mysql-