Clustering fails on clean environment

[Marco Filipe Moutinho da Silva]

Clustering doesn't always work on a clean environment. This was deployed using this charm on a clean environment (OpenStack): cs:~openstack-charmers-next/mysql-innodb-cluster-31

LOGS
====
unit-mysql-innodb-cluster-2: 14:55:05 INFO unit.mysql-innodb-cluster/2.juju-log Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:61:joined:cluster
unit-mysql-innodb-cluster-1: 14:55:06 DEBUG unit.mysql-innodb-cluster/1.juju-log Running _assess_status()
unit-mysql-innodb-cluster-2: 14:55:06 INFO unit.mysql-innodb-cluster/2.juju-log Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:66:changed:cluster
unit-mysql-innodb-cluster-2: 14:55:06 DEBUG unit.mysql-innodb-cluster/2.juju-log Running _assess_status()
unit-mysql-innodb-cluster-0: 14:55:07 ERROR unit.mysql-innodb-cluster/0.juju-log cluster:1: Failed adding instance 10.5.0.19 to cluster: Logger: Tried to log to an uninitialized logger.
WARNING: A GTID set check of the MySQL instance at '10.5.0.19:3306' determined that it
contains transactions that do not originate from the cluster, which must be
discarded before it can join the cluster.

10.5.0.19:3306 has the following errant GTIDs that do not exist in the cluster:
e12d3fa8-8e16-11ea-a5ae-fa163eca5945:1-16

WARNING: Discarding these extra GTID events can either be done manually or by completely
overwriting the state of 10.5.0.19:3306 with a physical snapshot from an
existing cluster member. To use this method by default, set the
'recoveryMethod' option to 'clone'.
====

Attached is the juju crash-dump

[David Ames (thedac)]

TRIAGE:

This may be a trailing java script style setting for "recoveryMethod" [0] rather than the pythonic "recovery_method."

Look for any other possible java script camel capitalized variables or settings and change them to pythonic underscore separated "_".

[0] https://github.com/openstack/charm-mysql-innodb-cluster/blob/master/src/lib/charm/openstack
/mysql_innodb_cluster.py#L650

[David Ames (thedac)]

Actually, this is due to a new version of mysql 8.0.20-0ubuntu0.20.04.1.

Still investigating.

[David Ames (thedac)]

With the apt packages of mysql 8.0.20-0ubuntu0.20.04.1 the current snap version of mysql-shell segfaults on add_instance().

A rebuilt snap using the latest of the 8.0 branches for mysql-shell and mysql-server works.

This is a msyql-shell bug not a mysql-innodb-cluster bug.

[David Ames (thedac)]

mysql-shell [0] now has several "collaborators".

The ~snap-mysql-shell-developers team [1] has successful builds of the snap on several architectures [2].

The snap store has the new version:

name: mysql-shell
summary: MySQL Shell
publisher: Canonical✓
store-url: https://snapcraft.io/mysql-shell
license: unset
description: |
  MySQL Shell (part of MySQL Server) 8.0
  MySQL query and administration shell client and framework.
snap-id: BW9WDw0mGu96vt1MszgZPKgiPw1W1aB4
channels:
  latest/stable: –
  latest/candidate: –
  latest/beta: –
  latest/edge: 8.0.20 2020-05-05 (4) 83MB -

mysql-innodb-cluster smoke tests pass using the 8.0.20 version of the snap from the snap store. Focal testing should be unblocked.

[0] https://snapcraft.io/mysql-shell
[1] https://launchpad.net/~snap-mysql-shell-developers
[2] https://code.launchpad.net/~snap-mysql-shell-developers/+snap/mysql-shell