mysql-router charm blocked with status 'hook failed: "update-status"'

[Jeffrey Chang]

From SQA testrun https://solutions.qa.canonical.com/testruns/3f475cc8-9dd0-4323-a3a2-f1a5c57ae11f
We see a new mysql-router charm rev 199 promoted on Jun 17.
Artifacts - https://oil-jenkins.canonical.com/artifacts/3f475cc8-9dd0-4323-a3a2-f1a5c57ae11f/index.html

On several units that has mysql-router charms, barbican/keystone/placement/vault/gnnochi, we saw 'hook failed: "update-status"'
But aodh/cinder/designate/glance/heat are ok.

Debug log shows
2024-06-17 23:14:14 ERROR unit.vault-mysql-router/1.juju-log server.go:325 Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 379, in dispatch
    _invoke(hook_handlers)
  File "/var/lib/juju/agents/unit-vault-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-mysql-router-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-mysql-router-1/charm/reactive/mysql_router_handlers.py", line 120, in update_status
    instance.validate_configuration()
  File "/var/lib/juju/agents/unit-vault-mysql-router-1/charm/lib/charm/openstack/mysql_router.py", line 491, in validate_configuration
    conf_size = os.path.getsize(self.mysqlrouter_conf)
  File "/usr/lib/python3.10/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/mysql/vault-mysql-router/mysqlrouter.conf'

[DUFOUR Olivier (odufourc)]

I can confirm I'm seeing this issue as well in my current deployment.

But the type of application doesn't matter, in some cases I have for example 2 units of aodh in error, and one being fine compared to the others.

I noticed on the problematic units that '/var/lib/mysql/' is currently empty.

[DUFOUR Olivier (odufourc)]

I'm under the impression that after running a hook, the entire configuration in '/var/lib/mysql' ends up being deleted and then the unit ends up in error.

I'm attaching the juju logs of one unit.

But the reproducer should be simple, do a deployment with multiple mysql-router units, and some of them may end up in error.

[Jadon Naas (jadonn)]

This bug was introduced in a recent change (https://review.opendev.org/c/openstack/charm-mysql-router/+/921716/2/src/reactive/mysql_router_handlers.py). We will work on reverting that change to fix this.

[Felipe Reyes (freyes)]

z-o-t revert - https://github.com/openstack-charmers/zaza-openstack-tests/pull/1224

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/922262

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/922263

[Seyeong Kim (seyeongkim)]

uploaded patch related to this.
could you please also consider not just reverting the original one but also the patch?
as reverting also took a time at least.(until my timezone..)
Thanks a lot.

https://review.opendev.org/c/openstack/charm-mysql-router/+/922262

[Alex Kavanagh (ajkavanagh)]

> uploaded patch related to this.
> could you please also consider not just reverting the original one but also the patch?
> as reverting also took a time at least.(until my timezone..)

We have to revert the patch as if we don't it would mean the charm is (deliberately) broken at that point in the git history; reverting the patch makes the repo 'clean' again. The correct thing to do, is to now rework the original patch so that it doesn't have the breaking behaviour *and* improve the testing such that the race-hazard with update-status is covered.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-router/+/922229
Committed: https://opendev.org/openstack/charm-mysql-router/commit/56165afebc77e130ad831d41dba7d31db86336f1
Submitter: "Zuul (22348)"
Branch: stable/jammy

commit 56165afebc77e130ad831d41dba7d31db86336f1
Author: Jadon Naas <email address hidden>
Date: Tue Jun 18 15:31:34 2024 +0000

    Revert "Fixing broken mysql-router configuration"

    This reverts commit d9af34ac232e0dcb022298074ed6b91c7e4476a9.

    Reason for revert: This change is causing a critical failure in the mysql-router as documented in https://bugs.launchpad.net/charm-mysql-router/+bug/2069661.

    Closes-Bug: #2069661
    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1224
    Change-Id: I4b06bfc63fbee6755ef028715f318af5a727cd5a

[Felipe Reyes (freyes)]

fix in revision 200

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-router/+/922228
Committed: https://opendev.org/openstack/charm-mysql-router/commit/38249fff87273f5dda56a64e84a421a03c4052ff
Submitter: "Zuul (22348)"
Branch: master

commit 38249fff87273f5dda56a64e84a421a03c4052ff
Author: Jadon Naas <email address hidden>
Date: Tue Jun 18 15:30:19 2024 +0000

    Revert "Fixing broken mysql-router configuration"

    This reverts commit d334665b2d8bb113223508f348d053020e8c860c.

    Reason for revert: This change is causing a critical failure in the mysql-router as documented in https://bugs.launchpad.net/charm-mysql-router/+bug/2069661.

    Closes-Bug: #2069661
    Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/1225
    Change-Id: I562293e24a6c8a45d49463d1a474e8af2bb0d5c5

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Seyeong Kim <email address hidden>" on branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/922263

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Seyeong Kim <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/922262