upgrading percona cluster from ~15.04 to 17.11 fails while mistakenly attempting to rebootstrap the cluster

[Paul Collins]

Upgrading from percona-cluster ~15.04 (r52, revision-id: <email address hidden>) to 17.11 fails as follows:

2018-02-01 03:26:12 INFO upgrade-charm * Stopping MySQL (Percona XtraDB Cluster) mysqld
2018-02-01 03:26:12 INFO upgrade-charm ...done.
2018-02-01 03:26:12 INFO upgrade-charm * Bootstrapping Percona XtraDB Cluster database server mysqld
2018-02-01 03:28:02 INFO upgrade-charm * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
2018-02-01 03:28:02 INFO upgrade-charm ...fail!
2018-02-01 03:28:02 INFO upgrade-charm Traceback (most recent call last):
[...]
2018-02-01 03:28:02 INFO upgrade-charm File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/upgrade-charm", line 221, in render_config_restart_on_changed
2018-02-01 03:28:02 INFO upgrade-charm bootstrap_pxc()
[...]
2018-02-01 03:28:02 INFO upgrade-charm File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
2018-02-01 03:28:02 INFO upgrade-charm raise child_exception
2018-02-01 03:28:02 INFO upgrade-charm OSError: [Errno 2] No such file or directory

From reading the code and comparing two deployments, it seems that this is due to is_bootstrapped() mistakenly returning False because bootstrap-uuid is not set on the cluster relation:

def is_bootstrapped():
    [...]
        for unit in units:
            id = relation_get('bootstrap-uuid', unit=unit, rid=rid)
            if id:
                uuids.append(id)
    [...]
    if uuids:
        [...]
        return True
    return False

$ juju run --unit percona-cluster/0 'relation-get -r cluster:4 bootstrap-uuid percona-cluster/0'
$ juju run --unit percona-cluster-seventeen-eleven/0 'relation-get -r cluster:16 bootstrap-uuid percona-cluster-seventeen-eleven/0 '
aabb5dad-0702-11e8-9b4d-ee83bd330150
$ _

[Pen Gale (pengale)]

This bug has been hanging on for a while without further activity, and the is_bootstrapped routine has been updated since the bug was filed. Marking this as Low, and I think the best plan is to set aside some time to verify that the issue no longer affects us, and close.

If this is affecting anybody actively upgrading charms, please leave a comment, and we'll bump the priority.