rabbitmq leader unit doesn't settle after changing source config

[Gabriel Cocenza]

I've observed that in a rabbitmq-server cluster, the leader units never settles when the "source" config is changed. This happened every time that I upgraded a cloud from focal-ussuri to focal-yoga.

Logs can be found here:
https://pastebin.canonical.com/p/Cy7qvJwHjg/

This required manual intervention to pause and resume the leader unit and in some cases ssh into the unit and reboot it.

[Alex Kavanagh (ajkavanagh)]

Please could you provide more information:

* a juju status for the model
* complete logs from the affected units.
* what the "source" config was changed from and to.

Thanks.

[Launchpad Janitor (janitor)]

[Expired for OpenStack RabbitMQ Server Charm because there has been no activity for 60 days.]

[Gabriel Cocenza (gabrielcocenza)]

Hi Alex. I'm attaching the logs from the rmq units that have the issue that I've described before. The change on configuration "source" was from "distro" to "cloud:focal-victoria. It"happened at 2024-04-15 18:52:24 to make easier to analyze the issue. Here is also a link[0] for the juju status as asked.

[0]https://pastebin.canonical.com/p/nwZrSWYGj6/

[Alex Kavanagh (ajkavanagh)]

Hi Gabriel;

Ok, so the problem is probably in the coordinator software in charmhelpers. It would be useful if you had a "juju show-unit" for each of the rabbitmq units as that might indicate why the locks are being held.

[Gabriel Cocenza (gabrielcocenza)]

Hi Alex, here is a pastebin with the juju show-units

https://pastebin.canonical.com/p/Yj44xRVx4X/

[Chi Wai CHAN (raychan96)]

I have the same issue. The steps to reproduce is outline here (using juju 3.5.0, and juju controller 3.3.1):

```
juju deploy rabbitmq-server --base ubuntu@20.04 -n 3
juju config rabbitmq-server enable-auto-restarts=True
juju config rabbitmq-server source=cloud:focal-victoria
```