post-series-upgrade hook: catch mysql_upgrade error

[Alvaro Uria]

I hit this bug when running OS Charms 19.04, on an upgrade from xenial to bionic.

post-series-upgrade hook can fail when "mysql_upgrade" was already run:
https://github.com/openstack/charm-percona-cluster/blob/stable/19.04/hooks/percona_hooks.py#L417

~# mysql_upgrade -p
Enter password:
Checking if update is needed.
This installation of MySQL is already upgraded to 5.7.20-18, use --force if you still need to run mysql_upgrade

subprocess.check_call should catch the output above and ignore the error instead of making the hook fail.

[vinaya (agrahar)]

does this going to fix ?

[James Page (james-page)]

We need to figure out why the mysql_upgrade is being executed twice - I believe it needs to be executed per unit to perform any required updates to the local db files which is why the mysqld is running in local mode during this step.

[Alex Kavanagh (ajkavanagh)]

I've tried reproducing this a couple of times (there's event a xenial->bionic upgrade bundle in percona-cluster now), but it doesn't seem to fail.

Is there any additional information you can provide as to what might have triggered the error?

[Xav Paice (xavpaice)]

Please see https://bugs.launchpad.net/charm-percona-cluster/+bug/1949935. The process was simple, just ran the upgrade on the leader unit first, one step at a time. The same error occurred on the remaining two percona-cluster units.

Is it something as simple as the percona cluster packages running the upgrade steps as well as the charm? Could the charm be modified to catch the exception and continue, since it's just the tool reporting that the upgrade step doesn't need to be run?

[Billy Olsen (billy-olsen)]

@xavpaice - would you be able to collect the logs from all of the percona units for further investigation?

[Liam Young (gnuoy)]

I have failed to reproduce this bug too. I've documented the steps i went through (based on https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/upgrade-series-openstack.html#generalised-openstack-series-upgrade ). This is the steps I went through https://pastebin.ubuntu.com/p/H8Zqvsb6DB/ . Please can you confirm that you followed the steps and as billy-olsen mentioned please provide logs.

[Peter Matulis (petermatulis)]

There is an error in the logic of the steps in the documentation. I'm not sure if this is the cause of the problem reported here, but these steps will soon be corrected.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-deployment-guide/+/818848
Committed: https://opendev.org/openstack/charm-deployment-guide/commit/a655378cca96f8f2e518eca6fa5a32cf88a4626b
Submitter: "Zuul (22348)"
Branch: master

commit a655378cca96f8f2e518eca6fa5a32cf88a4626b
Author: Peter Matulis <email address hidden>
Date: Tue Nov 16 16:26:43 2021 -0500

    Review series upgrades

    Review and correct the upgrade-series page.

    Make miscellaneous improvements to various places
    for the sake of consistency.

    Also closes a doc bug and reverts changes made due to
    a second doc bug.

    Closes-Bug: #1838041
    Related-Bug: #1934764
    Change-Id: I88692573e8ae50cd77d3872b40361b464b8e0f19