InconsistentUUIDError

[Louis Laporte]

When removing all of the mysql units (with juju remove-machine --force) the UUID of the old cluster master is remembered when adding new units.

Example:
$ juju remove-machine 0/lxd/6 1/lxd/6 2/lxd/6 --force
# wait for machines and units to be removed
$ juju add-unit mysql -n 3 --to lxd:0,lxd:1,lxd:2

In the log file /var/log/juju/unit-mysql-3.log the following message is present:

percona_utils.InconsistentUUIDError: Leader UUID ('9a348dd2-7690-11e8-b5fb-375fe856da9e') != Unit UUID ('c312623c-7a24-11e8-8f21-df9e90173fb9')

$ juju status mysql
App Version Status Scale Charm Store Rev OS Notes
hacluster-mysql blocked 1 hacluster jujucharms 46 ubuntu
mysql 5.6.37-26.21 error 3 percona-cluster jujucharms 266 ubuntu

Unit Workload Agent Machine Public address Ports Message
mysql/3 error idle 0/lxd/12 10.20.0.40 hook failed: "leader-settings-changed"
mysql/4 error idle 1/lxd/12 10.20.0.20 hook failed: "leader-settings-changed"
mysql/5* blocked idle 2/lxd/12 10.20.0.33 3306/tcp Insufficient peers to bootstrap cluster
  hacluster-mysql/3* blocked idle 10.20.0.33 Insufficient peer units for ha cluster (require 3)

# Commands to retrieve error
$ juju run --unit mysql/5 "cat /var/lib/percona-xtradb-cluster/rsync_sst_complete | cut -d: -f1"
c312623c-7a24-11e8-8f21-df9e90173fb9

$ juju run --unit mysql/5 "leader-get bootstrap-uuid"
9a348dd2-7690-11e8-b5fb-375fe856da9e

# Workaround
juju run --unit mysql/5 "leader-set bootstrap-uuid=c312623c-7a24-11e8-8f21-df9e90173fb9"

after that, all hacluster are up but mysql is never boostraped.

Bug related: https://bugs.launchpad.net/charm-percona-cluster/+bug/1742554

[Chris MacNaughton (chris.macnaughton)]

Given that this bug is requesting that removing all units and then adding new units be treated as a new charm, I'm re-targeting this to Juju and suggesting that it's a wishlist bug

[Richard Harding (rharding)]

I appreciate that this didn't work out as expected but if you want to redeploy something we're going to encourage actually redeploying as a new application. You can do that by providing an optional "name" argument to the deploy command.

    juju deploy mysql mysqlpart2

We don't want to have removing units turn into resetting application state as if it was a new deploy as that breaks the model.