upgrading charm does not trigger upgrade-charm hook properly

Bug #1993100 reported by DUFOUR Olivier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack RabbitMQ Server Charm
Fix Committed
Undecided
DUFOUR Olivier

Bug Description

It looks like "upgrade-charm" hook points to a bash script to handle python2 to python3 migration.

Within the shell script there is a line with :
```exec ./hooks/upgrade-charm.real```
https://opendev.org/openstack/charm-rabbitmq-server/src/commit/7c65a33ea8bb115b29fa3e44a3086534e5a0b66f/hooks/upgrade-charm#L21

However as seen in the logs, there is no hook ```upgrade-charm.real``` in rabbitmq_server_relations.py so it is skipped :
2022-10-17 02:32:18 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running upgrade-charm hook
2022-10-17 02:32:18 DEBUG juju.worker.uniter.runner runner.go:716 starting jujuc server {unix @/var/lib/juju/agents/unit-rabbit-focal-0/agent.socket <nil>}
2022-10-17 02:32:18 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
2022-10-17 02:32:18 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease
2022-10-17 02:32:19 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Hit:3 http://archive.ubuntu.com/ubuntu focal-security InRelease
2022-10-17 02:32:19 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
2022-10-17 02:32:19 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Get:5 https://esm.ubuntu.com/cis/ubuntu focal InRelease [3138 B]
2022-10-17 02:32:19 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Fetched 3138 B in 1s (2970 B/s)
2022-10-17 02:32:20 DEBUG unit.rabbit-focal/0.upgrade-charm logger.go:60 Reading package lists...
2022-10-17 02:32:20 INFO unit.rabbit-focal/0.juju-log server.go:316 coordinator.Serial Loading state
2022-10-17 02:32:20 INFO unit.rabbit-focal/0.juju-log server.go:316 coordinator.Serial New peer relation. Merging local state
2022-10-17 02:32:20 INFO unit.rabbit-focal/0.juju-log server.go:316 coordinator.Serial Leader handling coordinator requests
2022-10-17 02:32:20 INFO unit.rabbit-focal/0.juju-log server.go:316 Unknown hook upgrade-charm.real - skipping.

This has been confirmed on my side as happening on 3.9/stable branch and on master branch when working on an unrelated fix in the upgrade-charm path.
I've been able to re-enable that hook by renaming
```@hooks.hook('upgrade-charm')``` to ```@hooks.hook('upgrade-charm.real')```
https://opendev.org/openstack/charm-rabbitmq-server/src/commit/7c65a33ea8bb115b29fa3e44a3086534e5a0b66f/hooks/rabbitmq_server_relations.py#L774

But the hook would go in error with the block of code below.
    rabbit.update_peer_cluster_status()
    if not rabbit.clustered_with_leader():
        serial = coordinator.Serial()
        log("Requesting cluster join lock")
        serial.acquire(rabbit.COORD_KEY_CLUSTER)
        coordinated_cluster()

A quick workaround was to simply remove the block since it wasn't executed anyway with the entire hook being skipped so far.
One of the guess that code would fail, is related to the fact that rabbitmq is being stopped as part of the pre_install_hooks()

Changed in charm-rabbitmq-server:
assignee: nobody → DUFOUR Olivier (odufourc)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-rabbitmq-server (master)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-rabbitmq-server (master)

Reviewed: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/862480
Committed: https://opendev.org/openstack/charm-rabbitmq-server/commit/2c1074e49fe15e86eef958dd2820d008fc842ec4
Submitter: "Zuul (22348)"
Branch: master

commit 2c1074e49fe15e86eef958dd2820d008fc842ec4
Author: Olivier Dufour-Cuvillier <email address hidden>
Date: Tue Oct 25 11:53:16 2022 +0900

    Fix upgrade-charm hook that was only partly run

    upgrade-charm bash script was only executed and the hook
    from rabbitmq_server_relations.py was silently ignored
    This corrects at the same time a failure in the hook if
    rabbitmq deployment was from a single unit without any
    cluster.

    Closes-Bug: #1993100
    Change-Id: Iaf88d18d26a1cde23397a097fcd98b09a0a98846

Changed in charm-rabbitmq-server:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.