Fail to connect to MySQL after upgrade that includes PID change

[Rodrigo Barbieri]

Opening a new bug to track the current state of the charm for the purpose of fixing the original issue [1].

Basically when upgrading from an older version of the charm (such as charm store revision 6) that does not include commit [2], which updates the /etc/systemd/system/{}-mysql-router.service file definition and adds the PID file path to the config file, the systemd definition is updated only when upgrade_charm method finishes running, as the template is re-rendered.

After the many recently-landed patches up to [3], what happens now is that the config_changed hook is invoked at [4], before upgrade-charm is run, and then it writes a new config file and attempts to restart at [5], which does not succeed because upgrade-charm hasn't run and the systemd definition was not updated yet.

What needs to happen is that the restart [5] is skipped, so upgrade_charm runs first, then [5] runs. Aside from the PID change, the currently flow is also not correct as [6] makes adjustments to the config and needs to be run before, and it is currently running after [5].

[1] https://bugs.launchpad.net/charm-mysql-router/+bug/1927981
[2] https://github.com/openstack/charm-mysql-router/commit/72aca90ea3551a95ecf0bf301fd626bfb8b25a97
[3] https://review.opendev.org/c/openstack/charm-mysql-router/+/841708
[4] https://github.com/openstack/charm-mysql-router/blob/a49fc1e693f7b53526e9286e4d2d289f7e635d4e/src/reactive/mysql_router_handlers.py#L102
[5] https://github.com/openstack/charm-mysql-router/blob/a49fc1e693f7b53526e9286e4d2d289f7e635d4e/src/lib/charm/openstack/mysql_router.py#L781
[6] https://github.com/openstack/charm-mysql-router/blob/a49fc1e693f7b53526e9286e4d2d289f7e635d4e/src/lib/charm/openstack/mysql_router.py#L337

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/848673

[Felipe Reyes (freyes)]

On Mon, 2022-07-04 at 20:21 +0000, Rodrigo Barbieri wrote:
> Public bug reported:
>
> Opening a new bug to track the current state of the charm for the
> purpose of fixing the original issue [1].
>
> Basically when upgrading from an older version of the charm (such as
> charm store revision 6) that does not include commit [2], which updates
> the /etc/systemd/system/{}-mysql-router.service file definition and adds
> the PID file path to the config file, the systemd definition is updated
> only when upgrade_charm method finishes running, as the template is re-
> rendered.
>
> After the many recently-landed patches up to [3], what happens now is
> that the config_changed hook is invoked at [4], before upgrade-charm is
> run, and then it writes a new config file and attempts to restart at
> [5], which does not succeed because upgrade-charm hasn't run and the
> systemd definition was not updated yet.

I have this patch in the queue since we detected recently that on upgrade-charm
we are not restarting the mysql-router service.

https://review.opendev.org/c/openstack/charm-mysql-router/+/848748

Currently we only restart on changes to mysqlrouter.conf , even when we change
the file /etc/systemd/system/*-mysql-router.service , we should probably address
that.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Felipe, the systemd update due to PID change is currently performed during the parent upgrade-charm function run. That code needs to be run to solve the error of the mysql-router service being stuck if restarted earlier, that is currently being done in config-changed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-router/+/848673
Committed: https://opendev.org/openstack/charm-mysql-router/commit/6df798a939c9f5e167a73f6e8c145c6ec631ac6c
Submitter: "Zuul (22348)"
Branch: master

commit 6df798a939c9f5e167a73f6e8c145c6ec631ac6c
Author: Rodrigo Barbieri <email address hidden>
Date: Mon Jul 4 17:22:02 2022 -0300

    Fix restarts during upgrade-charm hook

    Upgrade-charm code needs to make various adjustments to
    the config files and restart the mysql-router service,
    but the code is currently running config-changed and
    getting stuck in a restart loop before upgrade-charm
    is able to run to make the config adjustments and
    invoke a successful restart.

    This change adds a condition to skip the execution of
    config_changed function when upgrade-charm hook runs,
    so upgrade_charm code takes care of the adjustments.

    Also, now upgrade-charm hook is guaranteed to restart the
    mysql-router service whenever any change to the config
    file is made as part of its execution, whereas before
    it wasn't doing so when making the adjustments of the
    custom upgrade_charm function, resulting in the possibility
    of no restarts after a charm upgrade.

    This change includes the now-abandoned change:
    https://review.opendev.org/848748

    Co-Authored-By: Felipe Reyes <email address hidden>

    Closes-bug: #1980693
    Closes-bug: #1979263
    Related-bug: #1927981
    Change-Id: If9d71bdb839c9c0ee3f4b33e4d44a5c93bdd13de

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/jammy
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/851761

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-router/+/851761
Committed: https://opendev.org/openstack/charm-mysql-router/commit/cfe6b2e3c0081fdc9c3aafec26a942e722de34a9
Submitter: "Zuul (22348)"
Branch: stable/jammy

commit cfe6b2e3c0081fdc9c3aafec26a942e722de34a9
Author: Rodrigo Barbieri <email address hidden>
Date: Mon Jul 4 17:22:02 2022 -0300

    Fix restarts during upgrade-charm hook

    Upgrade-charm code needs to make various adjustments to
    the config files and restart the mysql-router service,
    but the code is currently running config-changed and
    getting stuck in a restart loop before upgrade-charm
    is able to run to make the config adjustments and
    invoke a successful restart.

    This change adds a condition to skip the execution of
    config_changed function when upgrade-charm hook runs,
    so upgrade_charm code takes care of the adjustments.

    Also, now upgrade-charm hook is guaranteed to restart the
    mysql-router service whenever any change to the config
    file is made as part of its execution, whereas before
    it wasn't doing so when making the adjustments of the
    custom upgrade_charm function, resulting in the possibility
    of no restarts after a charm upgrade.

    This change includes the now-abandoned change:
    https://review.opendev.org/848748

    Co-Authored-By: Felipe Reyes <email address hidden>

    Closes-bug: #1980693
    Closes-bug: #1979263
    Related-bug: #1927981
    Change-Id: If9d71bdb839c9c0ee3f4b33e4d44a5c93bdd13de
    (cherry picked from commit 6df798a939c9f5e167a73f6e8c145c6ec631ac6c)