Scale-in: cluster-relation-departed and db-router departed hook error

[Andrew McLeod]

unit-mysql-innodb-cluster-2: 15:57:02 INFO unit.mysql-innodb-cluster/2.juju-log db-router:54: Invoking reactive handler: hooks/relations/mysql-router/provides.py:66:departed:db-router
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG worker.uniter.jujuc running hook tool "juju-log"
unit-mysql-innodb-cluster-2: 15:57:03 ERROR unit.mysql-innodb-cluster/2.juju-log db-router:54: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
TypeError: departed() missing 1 required positional argument: 'self'

unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed Traceback (most recent call last):
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/hooks/db-router-relation-departed", line 22, in <module>
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed main()
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed bus.dispatch(restricted=restricted_mode)
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-mysql-innodb-cluster-2: 15:57:03 DEBUG unit.mysql-innodb-cluster/2.db-router-relation-departed _invoke(other_handlers)
unit-mysql-innodb-cluster-2: 15:57:15 DEBUG juju.worker.uniter.remotestate update status timer triggered
unit-mysql-innodb-cluster-2: 15:57:15 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook

[Andrew McLeod (admcleod)]

cross-ref https://bugs.launchpad.net/charm-mysql-router/+bug/1874480

[Marco Filipe Moutinho da Silva (mfmsilva)]

Same issue with cluster-relation-departed hook.
Part of the logs:

unit-mysql-innodb-cluster-2: 14:25:01 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:95:departed:cluster
unit-mysql-innodb-cluster-1: 14:25:01 ERROR unit.mysql-innodb-cluster/1.juju-log cluster:0: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
TypeError: departed() missing 1 required positional argument: 'self'

[David Ames (thedac)]

TRIAGE:

The use of when_any for the peer interface handler is the root cause:

class MySQLInnoDBClusterPeer(reactive.Endpoint):

...

    @reactive.when_any('endpoint.{endpoint_name}.broken',
                       'endpoint.{endpoint_name}.departed')
    def departed(self):

[David Ames (thedac)]

This is a scale in problem when removing a node from the cluster.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728582

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728585

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728587

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728580
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-router/commit/?id=ffd0fc0dc48ed9d85911894a8038397ddce8ef69
Submitter: Zuul
Branch: master

commit ffd0fc0dc48ed9d85911894a8038397ddce8ef69
Author: David Ames <email address hidden>
Date: Fri May 15 14:32:56 2020 -0700

    Scale-in fixes

    Properly handle departed and broken hooks. Keep correct state during
    both scale in and scale out.

    Change-Id: Ibc87371f441318e2082ff4028d9f57ae6b796035
    Closes-Bug: #1874479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728585
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-innodb-cluster/commit/?id=5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Submitter: Zuul
Branch: master

commit 5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Author: David Ames <email address hidden>
Date: Fri May 15 15:17:59 2020 -0700

    Scale-in and Scale-out of mysql-innodb-cluster

    Handle adding and removing units to the mysql-innodb-cluster.

    Change-Id: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Closes-Bug: #1877546
    Closes-Bug: #1874479

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728582
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-router/commit/?id=05525dad0270c9a9a386e81202879ef6ad5d5688
Submitter: Zuul
Branch: master

commit 05525dad0270c9a9a386e81202879ef6ad5d5688
Author: David Ames <email address hidden>
Date: Fri May 15 14:43:15 2020 -0700

    Scale-in of mysql innodb cluster nodes

    Handle departed and broken db-router relations elegantly.
    Remove Eoan as a test target.

    Closes-Bug: #1874479
    Depends-On: Ibc87371f441318e2082ff4028d9f57ae6b796035
    Change-Id: Ida5e8fef458abc38cc50ad8749f315eadcb499e0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728587
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-innodb-cluster/commit/?id=7cbf9a88b001d73d290bb123f5bba010f51319b3
Submitter: Zuul
Branch: master

commit 7cbf9a88b001d73d290bb123f5bba010f51319b3
Author: David Ames <email address hidden>
Date: Wed May 13 11:37:31 2020 -0700

    Scale in and scale out

    Handle adding and removing units of the mysql-innodb-cluster gracefully.

    Review and land:
    https://github.com/openstack-charmers/zaza/pull/353
    https://github.com/openstack-charmers/zaza-openstack-tests/pull/275

    Closes-Bug: #1874479
    Closes-Bug: #1877546
    Closes-Bug: #1877616
    Depends-On: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Change-Id: I39210c760e7204f84365d5dcb4eeedbc0889041f