expansion of single unit deployment fails

Bug #1756308 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Percona Cluster Charm
Fix Released
High
David Ames

Bug Description

Single unit deployment; configured min-cluster-size to 3 and then added two new units. Both installed and then the leader-settings-changed hook failed - rendering of the configuration files is driven based on uuid's presented on the cluster relation, which at that point in time was not visible.

I think the right solution is to use the leader storage provided value.

Revision history for this message
James Page (james-page) wrote :

Or maybe its actually correct to defer writing the configuration file at this point, and try later once the cluster relation has been joined.

Revision history for this message
James Page (james-page) wrote :

2018-03-16 10:58:08 ERROR juju-log bootstrap uuid differs: 335e9d81-2903-11e8-a978-2acd66baf54f != de5cb2a1-2908-11e8-be08-deac84da496c
2018-03-16 10:58:08 DEBUG leader-settings-changed Traceback (most recent call last):
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 859, in <module>
2018-03-16 10:58:08 DEBUG leader-settings-changed main()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 849, in main
2018-03-16 10:58:08 DEBUG leader-settings-changed hooks.execute(sys.argv)
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-16 10:58:08 DEBUG leader-settings-changed self._hooks[hook_name]()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 815, in leader_settings_changed
2018-03-16 10:58:08 DEBUG leader-settings-changed config_changed()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-03-16 10:58:08 DEBUG leader-settings-changed return f(*args, **kwargs)
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/leader-settings-changed", line 406, in config_changed
2018-03-16 10:58:08 DEBUG leader-settings-changed update_bootstrap_uuid()
2018-03-16 10:58:08 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-mysql-4/charm/hooks/percona_utils.py", line 560, in update_bootstrap_uuid
2018-03-16 10:58:08 DEBUG leader-settings-changed cluster_state_uuid)
2018-03-16 10:58:08 DEBUG leader-settings-changed percona_utils.InconsistentUUIDError: Leader UUID ('335e9d81-2903-11e8-a978-2acd66baf54f') != Unit UUID ('de5cb2a1-2908-11e8-be08-deac84da496c')

Revision history for this message
James Page (james-page) wrote :

I think this is only revealed when switching a single unit into a clustered deployment - in a new deployment, the lead unit won't bootstrap until all other units have appeared.

James Page (james-page)
Changed in charm-percona-cluster:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → James Page (james-page)
Revision history for this message
David Ames (thedac) wrote :

Stealing this bug from James. While working on https://bugs.launchpad.net/charm-percona-cluster/+bug/1756928 this is being addressed.

See https://review.openstack.org/#/c/555494/

Changed in charm-percona-cluster:
assignee: James Page (james-page) → David Ames (thedac)
Changed in charm-percona-cluster:
status: Triaged → In Progress
Ryan Beisner (1chb1n)
Changed in charm-percona-cluster:
milestone: none → 18.05
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-percona-cluster (master)

Reviewed: https://review.openstack.org/555494
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=801c2e78294f7af0a058aca036b6d066e3f53a98
Submitter: Zuul
Branch: master

commit 801c2e78294f7af0a058aca036b6d066e3f53a98
Author: David Ames <email address hidden>
Date: Mon Mar 19 11:25:56 2018 -0700

    Redesign cluster buildup process

    In order to fix bug#1756928 the whole cluster buildup process needed to
    be redesigned. The assumptions about what is_bootstrapped and clustered
    meant and when to restart on configuration changed needed to be
    re-evaluated.

    The timing of restarts needed to be protected to avoid collisions.
    Only bootstrapped hosts should go in to the
    wsrep_cluster_address=gcomm:// setting. Adding or removing units should
    be handled gracefully. Starting with a single unit and expanding to a
    cluster must work.

    This change guarantees mysqld is restarted when the configuration file
    changes and meets all the above requirements. As a consequence of the redesign,
    the workload status now more accurately reflects the state of the unit.

    Charm-helpers sync to bring in distributed_wait fix.

    Closes-Bug: #1756308
    Closes-Bug: #1756928
    Change-Id: I0742e6889b32201806cec6a0b5835e11a8027567

Changed in charm-percona-cluster:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-percona-cluster (master)

Change abandoned by James Page (<email address hidden>) on branch: master
Review: https://review.openstack.org/553755

David Ames (thedac)
Changed in charm-percona-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.