coordinator-relation-changed hook error when pausing a unit after setting action-managed-upgrade

[Aurelien Lourot]

Steps to reproduce:
juju config mysql-innodb-cluster action-managed-upgrade=true
juju run-action mysql-innodb-cluster/0 pause

Expected result:
No hook error.

Actual result:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/reactive/mysql_innodb_cluster_handlers.py", line 302, in post_rolling_restart_update_clients
    db_router_respond()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/reactive/mysql_innodb_cluster_handlers.py", line 340, in db_router_respond
    if instance.create_databases_and_users(db_router):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1265, in create_databases_and_users
    allowed_units = self.get_allowed_units(
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1218, in get_allowed_units
    allowed_units = db_helper.get_allowed_units(
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charmhelpers/contrib/database/mysql.py", line 446, in get_allowed_units
    self.connect(password=self.get_mysql_root_password())
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
    return f(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charmhelpers/contrib/database/mysql.py", line 116, in connect
    self.connection = MySQLdb.connect(**_connection_info)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/MySQLdb/__init__.py", line 130, in Connect
    return Connection(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 185, in __init__
    super().__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")

Analysis:

1. One issue is that post_rolling_restart_update_clients shouldn't get called after pausing a unit. This reactive handler is designed to answer the clients after a rolling restart of the units after the applications got data over the certificates relation. This handler assumes that the local mysql is running, which isn't the case after having run a `pause` action.
2. Another issue is that probably this charm shouldn't have the action-managed-upgrade in the first place, since this feature is only relevant for OpenStack components on which you may want to perform a release upgrade by pointing the charm to an Ubuntu Cloud Archive.

Charm version:

#85 [1] improved with some custom robustness/retries around connection to mysql [2]

[1] https://jaas.ai/u/openstack-charmers-next/mysql-innodb-cluster
[2] https://github.com/juju/charm-helpers/pull/642

[Aurelien Lourot (aurelien-lourot)]

In a reactive charm, the `openstack-upgrade` action calls run_upgrade() [1] which is implemented around the concept of OpenStack release, which isn't appropriate for such an OpenStack-independant charm. The git history of this charm shows no intention of using this feature in this charm. This feature got inherited "by accident" from the openstack-principle reactive layer. [2]

[1] https://github.com/openstack/charms.openstack/blob/master/charms_openstack/charm/core.py#L1133
[2] https://github.com/openstack/charm-layer-openstack-principle/blob/master/actions.yaml

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/810866

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/810866
Committed: https://opendev.org/openstack/charm-mysql-innodb-cluster/commit/1774de95b7d55616b25f09b199f48f98fb6c2c32
Submitter: "Zuul (22348)"
Branch: master

commit 1774de95b7d55616b25f09b199f48f98fb6c2c32
Author: Aurelien Lourot <email address hidden>
Date: Wed Sep 22 09:52:48 2021 +0200

    Remove the openstack-upgrade action

    The openstack-principle reactive layer brings both the
    openstack-origin option and the openstack-upgrade
    action, none of which being needed for this charm.
    This layer has been used already since the very first
    commit of this charm, most likely for no specific
    reason.

    Also synced tox/pip boilerplate in order to include:
    * https://github.com/openstack-charmers/release-tools/pull/156
    * https://github.com/openstack-charmers/release-tools/pull/157

    Closes-Bug: #1944953
    Change-Id: I001f435c648229b65fb718a78a33e6303a0faf11