expansion of single unit deployment fails

[James Page]

Single unit deployment; configured min-cluster-size to 3 and then added two new units. Both installed and then the leader-settings-changed hook failed - rendering of the configuration files is driven based on uuid's presented on the cluster relation, which at that point in time was not visible.

I think the right solution is to use the leader storage provided value.

[James Page (james-page)]

Or maybe its actually correct to defer writing the configuration file at this point, and try later once the cluster relation has been joined.

[James Page (james-page)]

2018-03-16 10:58:08 ERROR juju-log bootstrap uuid differs: 335e9d81-2903-11e8-a978-2acd66baf54f != de5cb2a1-2908-11e8-be08-deac84da496c
2018-03-16 10:58:08 DEBUG leader-settings-changed Traceback (most recent call last):
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 859, in <module>
2018-03-16 10:58:08 DEBUG leader-settings-changed main()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 849, in main
2018-03-16 10:58:08 DEBUG leader-settings-changed hooks.execute(sys.argv)
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-16 10:58:08 DEBUG leader-settings-changed self._hooks[hook_name]()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 815, in leader_settings_changed
2018-03-16 10:58:08 DEBUG leader-settings-changed config_changed()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-03-16 10:58:08 DEBUG leader-settings-changed return f(*args, **kwargs)
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 406, in config_changed
2018-03-16 10:58:08 DEBUG leader-settings-changed update_bootstrap_uuid()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/percona_utils.py", line 560, in update_bootstrap_uuid
2018-03-16 10:58:08 DEBUG leader-settings-changed cluster_state_uuid)
2018-03-16 10:58:08 DEBUG leader-settings-changed percona_utils.InconsistentUUIDError: Leader UUID ('335e9d81-2903-11e8-a978-2acd66baf54f') != Unit UUID ('de5cb2a1-2908-11e8-be08-deac84da496c')

[James Page (james-page)]

I think this is only revealed when switching a single unit into a clustered deployment - in a new deployment, the lead unit won't bootstrap until all other units have appeared.

[David Ames (thedac)]

Stealing this bug from James. While working on https://bugs.launchpad.net/charm-percona-cluster/+bug/1756928 this is being addressed.

See https://review.openstack.org/#/c/555494/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555494
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=801c2e78294f7af0a058aca036b6d066e3f53a98
Submitter: Zuul
Branch: master

commit 801c2e78294f7af0a058aca036b6d066e3f53a98
Author: David Ames <email address hidden>
Date: Mon Mar 19 11:25:56 2018 -0700

    Redesign cluster buildup process

    In order to fix bug#1756928 the whole cluster buildup process needed to
    be redesigned. The assumptions about what is_bootstrapped and clustered
    meant and when to restart on configuration changed needed to be
    re-evaluated.

    The timing of restarts needed to be protected to avoid collisions.
    Only bootstrapped hosts should go in to the
    wsrep_cluster_address=gcomm:// setting. Adding or removing units should
    be handled gracefully. Starting with a single unit and expanding to a
    cluster must work.

    This change guarantees mysqld is restarted when the configuration file
    changes and meets all the above requirements. As a consequence of the redesign,
    the workload status now more accurately reflects the state of the unit.

    Charm-helpers sync to bring in distributed_wait fix.

    Closes-Bug: #1756308
    Closes-Bug: #1756928
    Change-Id: I0742e6889b32201806cec6a0b5835e11a8027567

[OpenStack Infra (hudson-openstack)]

Change abandoned by James Page (<email address hidden>) on branch: master
Review: https://review.openstack.org/553755