When testing Percona 5.7 packaging on Bionic with the master branch of this charm, I regularly hit "bootstrap uuid differs" failures on non-leader units. These occurred during the leader-settings-changed hook. leader_settings_changed() calls update_bootstrap_uuid(), where the InconsistentUUIDError exception is raised.
Looking through the charm code, non-leaders were getting False returned from the is_bootstrapped() call in config_changed(). Because of this, these units weren't rendering config, resulting in running with the default versions of /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf. Templated versions of mysqld.cnf need to be rendered in order for successful clustering (ie. setting up wsrep_sst_auth, wsrep_cluster_name, wsrep_cluster_address and more need to be set correctly). This was happening because the non-leaders had no 'bootstrap-uuid' set over their cluster relations.
Looking at the leader unit, it does set bootstrap-uuid via relation_set() for all relation IDs as well as via leader_set() in notify_bootstrapped(). Also worth noting, leader_set() is called last.
It appears that non-leaders get into the leader-settings-changed hook too early, before configs are rendered.
One thought for fixing this was to add an except for InconsistentUUI DError in leader_ settings_ changed( ) to handle the exception from update_ bootstrap_ uuid(). For example:
try: bootstrap_ uuid() apUUIDError: set('waiting' , "Waiting for bootstrap-uuid set by leader") DError: set('waiting' , "config-changed hook likely hasn't rendered config yet")
update_
except LeaderNoBootstr
status_
except InconsistentUUI
status_
The problem with this approach is that the leader- settings- changed hook might not fire again, and update_ bootstrap_ uuid() may not successfully get called.