after remove/add mysql units, unit becomes blocked with 'shared-db' missing

[Xav Paice]

Charm 18.08, Juju 2.4, Xenial and Queens.

Deployed 3 units of mysql, 3 of Gnocchi, aodh and designate, added relations, all was good. All use hacluster with a VIP set.

Added 2 new units of mysql (mysql/3, mysql/4)
Removed 2 older units of mysql (mysql/1 and mysql/2)

Now the gnocchi, aodh and Designate units sit with Workload "blocked" and the message "'shared-db' missing".

From the logs, it looks like the relation-departed hooks have set the status incorrectly, as there's still suitable database config in the gnocchi.conf.

2018-10-24 04:34:43 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-changed" hook
2018-10-24 04:34:43 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-changed (91; mysql/3) hook"

2018-10-24 04:39:39 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-changed" hook
2018-10-24 04:39:39 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-changed (91; mysql/4) hook"

2018-10-24 05:02:31 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-departed" hook
2018-10-24 05:02:31 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-departed (91; mysql/1) hook"

2018-10-24 05:02:37 INFO juju.worker.uniter.operation runhook.go:120 ran "shared-db-relation-departed" hook
2018-10-24 05:02:37 DEBUG juju.worker.uniter.operation executor.go:91 committing operation "run relation-departed (91; mysql/2) hook"

[David Ames (thedac)]

The interface-mysql-shared may need to unset the following on shared-db departed:

shared-db.available
shared-db.connected
db.synched

Triage:

Handle the case:
juju remove-relation $REACTIVE_CHARM mysql

Determine if the departed hook is the last unit in the shared-db relation
If so, unset all flags pertaining to the shared-db relation

[David Ames (thedac)]

In progress https://review.openstack.org/#/c/618601/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618601
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-shared/commit/?id=efa7cb3d5a81ab65d9cbf40e47ada5ec670a8a01
Submitter: Zuul
Branch: master

commit efa7cb3d5a81ab65d9cbf40e47ada5ec670a8a01
Author: David Ames <email address hidden>
Date: Fri Nov 16 11:43:35 2018 -0800

    Check if any HA units remain on departed

    If there are still HA units in the relation when a single unit departs
    re-evaluate state. If it is the last unit, unset all relation states.

    Handle the case juju remove-relation and juju add-relation.

    Change-Id: I92909d6203fc00d45d7d6edf0a981f9d759add53
    Partial-Bug: #1800730

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/620125

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/620127

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/620128

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620127
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=06febb43ecdf7e56b835418d02cf62841024516d
Submitter: Zuul
Branch: master

commit 06febb43ecdf7e56b835418d02cf62841024516d
Author: David Ames <email address hidden>
Date: Mon Nov 26 10:21:28 2018 -0800

    Rebuild reactive charms for mysql-shared interface

    Pull in the fix for Bug #1800730 in the mysql-shared interface.

    Change-Id: I3bd7c167678be2b27aa7597cf5e161b9a21d3689
    Partial-Bug: #1800730

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620125
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=78a28efa1482c6a321685b1458f9548b802698e7
Submitter: Zuul
Branch: master

commit 78a28efa1482c6a321685b1458f9548b802698e7
Author: David Ames <email address hidden>
Date: Mon Nov 26 10:21:13 2018 -0800

    Rebuild reactive charms for mysql-shared interface

    Pull in the fix for Bug #1800730 in the mysql-shared interface.

    Change-Id: If659b82eaf8b10e23066feb260c481ceb0208d23
    Partial-Bug: #1800730

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620128
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=77793a1a709bd54a9739d95392f93b2851edf315
Submitter: Zuul
Branch: master

commit 77793a1a709bd54a9739d95392f93b2851edf315
Author: David Ames <email address hidden>
Date: Mon Nov 26 10:21:35 2018 -0800

    Rebuild reactive charms for mysql-shared interface

    Pull in the fix for Bug #1800730 in the mysql-shared interface.

    Change-Id: I324afd953381dcca42b5a230078245ebd40196d1
    Partial-Bug: #1800730