after remove/add mysql units, unit becomes blocked with 'shared-db' missing

Bug #1800730 reported by Xav Paice on 2018-10-30
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Gnocchi Charm
Critical
David Ames
OpenStack AODH Charm
Critical
David Ames
OpenStack Designate Charm
Critical
David Ames
mysql-shared charm interface
Critical
David Ames

Bug Description

Charm 18.08, Juju 2.4, Xenial and Queens.

Deployed 3 units of mysql, 3 of Gnocchi, aodh and designate, added relations, all was good. All use hacluster with a VIP set.

Added 2 new units of mysql (mysql/3, mysql/4)
Removed 2 older units of mysql (mysql/1 and mysql/2)

Now the gnocchi, aodh and Designate units sit with Workload "blocked" and the message "'shared-db' missing".

From the logs, it looks like the relation-departed hooks have set the status incorrectly, as there's still suitable database config in the gnocchi.conf.

2018-10-24 04:34:43 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-changed" hook
2018-10-24 04:34:43 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-changed (91; mysql/3) hook"

2018-10-24 04:39:39 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-changed" hook
2018-10-24 04:39:39 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-changed (91; mysql/4) hook"

2018-10-24 05:02:31 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-departed" hook
2018-10-24 05:02:31 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-departed (91; mysql/1) hook"

2018-10-24 05:02:37 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-departed" hook
2018-10-24 05:02:37 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-departed (91; mysql/2) hook"

Xav Paice (xavpaice) on 2018-10-30
summary: - after remove/add mysql units, unit becomes blocked with'shared-db'
+ after remove/add mysql units, unit becomes blocked with 'shared-db'
missing
David Ames (thedac) wrote :

The interface-mysql-shared may need to unset the following on shared-db departed:

shared-db.available
shared-db.connected
db.synched

Triage:

Handle the case:
juju remove-relation $REACTIVE_CHARM mysql

Determine if the departed hook is the last unit in the shared-db relation
If so, unset all flags pertaining to the shared-db relation

Changed in charm-interface-mysql-shared:
status: New → Confirmed
importance: Undecided → Critical
milestone: none → 18.11
David Ames (thedac) wrote :
Changed in charm-interface-mysql-shared:
status: Confirmed → In Progress
David Ames (thedac) on 2018-11-20
Changed in charm-interface-mysql-shared:
milestone: 18.11 → 19.04

Reviewed: https://review.openstack.org/618601
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-shared/commit/?id=efa7cb3d5a81ab65d9cbf40e47ada5ec670a8a01
Submitter: Zuul
Branch: master

commit efa7cb3d5a81ab65d9cbf40e47ada5ec670a8a01
Author: David Ames <email address hidden>
Date: Fri Nov 16 11:43:35 2018 -0800

    Check if any HA units remain on departed

    If there are still HA units in the relation when a single unit departs
    re-evaluate state. If it is the last unit, unset all relation states.

    Handle the case juju remove-relation and juju add-relation.

    Change-Id: I92909d6203fc00d45d7d6edf0a981f9d759add53
    Partial-Bug: #1800730

David Ames (thedac) on 2018-11-26
Changed in charm-interface-mysql-shared:
assignee: nobody → David Ames (thedac)
status: In Progress → Fix Committed
David Ames (thedac) on 2018-11-26
Changed in charm-gnocchi:
status: New → Confirmed
Changed in charm-aodh:
status: New → Confirmed
Changed in charm-designate:
assignee: nobody → David Ames (thedac)
status: New → Confirmed
Changed in charm-gnocchi:
assignee: nobody → David Ames (thedac)
Changed in charm-aodh:
importance: Undecided → Critical
Changed in charm-designate:
importance: Undecided → Critical
Changed in charm-aodh:
assignee: nobody → David Ames (thedac)
Changed in charm-gnocchi:
importance: Undecided → Critical

Fix proposed to branch: master
Review: https://review.openstack.org/620125

Changed in charm-aodh:
status: Confirmed → In Progress
Changed in charm-designate:
status: Confirmed → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/620127

Changed in charm-gnocchi:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/620127
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=06febb43ecdf7e56b835418d02cf62841024516d
Submitter: Zuul
Branch: master

commit 06febb43ecdf7e56b835418d02cf62841024516d
Author: David Ames <email address hidden>
Date: Mon Nov 26 10:21:28 2018 -0800

    Rebuild reactive charms for mysql-shared interface

    Pull in the fix for Bug #1800730 in the mysql-shared interface.

    Change-Id: I3bd7c167678be2b27aa7597cf5e161b9a21d3689
    Partial-Bug: #1800730

Reviewed: https://review.openstack.org/620125
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=78a28efa1482c6a321685b1458f9548b802698e7
Submitter: Zuul
Branch: master

commit 78a28efa1482c6a321685b1458f9548b802698e7
Author: David Ames <email address hidden>
Date: Mon Nov 26 10:21:13 2018 -0800

    Rebuild reactive charms for mysql-shared interface

    Pull in the fix for Bug #1800730 in the mysql-shared interface.

    Change-Id: If659b82eaf8b10e23066feb260c481ceb0208d23
    Partial-Bug: #1800730

Reviewed: https://review.openstack.org/620128
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=77793a1a709bd54a9739d95392f93b2851edf315
Submitter: Zuul
Branch: master

commit 77793a1a709bd54a9739d95392f93b2851edf315
Author: David Ames <email address hidden>
Date: Mon Nov 26 10:21:35 2018 -0800

    Rebuild reactive charms for mysql-shared interface

    Pull in the fix for Bug #1800730 in the mysql-shared interface.

    Change-Id: I324afd953381dcca42b5a230078245ebd40196d1
    Partial-Bug: #1800730

David Ames (thedac) on 2018-11-29
Changed in charm-gnocchi:
status: In Progress → Fix Committed
Changed in charm-designate:
status: In Progress → Fix Committed
Changed in charm-aodh:
status: In Progress → Fix Committed
Changed in charm-gnocchi:
milestone: none → 19.04
Changed in charm-aodh:
milestone: none → 19.04
Changed in charm-designate:
milestone: none → 19.04
David Ames (thedac) on 2019-04-17
Changed in charm-gnocchi:
status: Fix Committed → Fix Released
Changed in charm-aodh:
status: Fix Committed → Fix Released
Changed in charm-designate:
status: Fix Committed → Fix Released
Changed in charm-interface-mysql-shared:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers