Cluster Rabbitmq with Kolla-Ansible Problem

[Khoi]

Description
===========
I set up 3 controllers and 3 compute nodes. My system cannot work well when 1 rabbit node in cluster rabbitmq is down, cannot launch instances. It stucked at scheduling.

Steps to reproduce
===========
Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672//
* Reboot 1 of 3 rabbitmq node.
* Create instances then it stucked at scheduling.

Workaround
===========
Point to rabbitmq VIP address. But We cannot share the load with this solution. Please give me some suggestions. Thank you very much.

I did google and enabled system log's debug but I still cannot understand why.

[Mesut Muhammet Şahin (messah)]

Hi,

Is there any update for Xena? When we upgrade our cluster from wallaby to xena, we also had the same problem.

I think this issue is related with https://bugs.launchpad.net/cloud-archive/+bug/1993149

This issue seems fixed in the openstack/oslo.messaging 12.13.1 release but our oslo.messaging version is 12.9.4 in services. How do we update oslo.messaging version for all services?

[Yusuf Güngör (yusuf2)]

hi everyone, there exist a bug on oslo.messaging which brokes the HA of rabbitmq. When rabbitmq leader unit is down then the consumers can not connect to other rabbitmq nodes.

There is a fix for oslo.messaging 12.9.4-1

But the installed oslo.messaging version is 12.9.4 for xena images. Can we update oslo.messaging to 12.9.4-1 for all images which uses oslo.messaging?

There is no problem for Yoga images, Yoga images has oslo.messaging 12.13.1 which has the patch.

$ docker pull quay.io/openstack.kolla/ubuntu-source-neutron-server:xena
$ docker run -it quay.io/openstack.kolla/ubuntu-source-neutron-server:xena bash
()[neutron@51cabaa9dc23 /]$ pip freeze
oslo.messaging==12.9.4

Sources:

- https://bugs.launchpad.net/kolla-ansible/+bug/1990257
- https://bugs.launchpad.net/cloud-archive/+bug/1993149
- https://docs.openstack.org/releasenotes/oslo.messaging/en_GB/xena.html

[Michal Nasiadka (mnasiadka)]

Xena is EM so those can't be updated.