oslo_messaging kombu strategy round-robin not working correctly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
oslo.messaging |
New
|
Undecided
|
Unassigned |
Bug Description
Hi,
We were doing some HA tests against our openstack cluster and we were testing what will happen if we turn off one rabbitmq from 3 node rabbitmq cluster and found that default oslo.messaging kombu_failover_
We turned off 10.157.106.71 and clients didn't reconnect. If I grepped occurences in logs for this rabbitmq server, i found that it is always trying that host which is turned off.
root@controller
5 10.157.106.136:5672
12 10.157.106.6:5672
50381 10.157.106.71:5672
root@controller
2 -]
6 10.157.106.136:5672
4 10.157.106.6:5672
41996 10.157.106.71:5672
I was also checking TCP SYN via netstat and I saw every time it was trying to connect to rabbitmq server which was down.
Here is video of cinder-volume which was always trying to connect to server which was down, normally it should try another rabbitmq server and connect.
How to reproduce ? Turn off one rabbitmq server and check logs.
I was testing on yoga version.