Killing one controller causes some OpenStack services stop consuming messages

[Dmitry Mescheryakov]

Version: 8.0

So far this bug was reproduced only when patch https://review.openstack.org/278440 is applied

Execute ha_destroy_controllers system test. The test fails on OSTF verification after 1 controller is destroyed. Several OSTF tests fail at the same time. Examining the setup reveals that several important queues are not listened by anyone:

root@node-2:~# rabbitmqctl list_queues consumers name | grep -v node-1
Listing queues ...
0 cinder-scheduler
0 dhcp_agent
0 l3_agent
0 q-agent-notifier-network-update
0 scheduler
1 cert.node-2.test.domain.local
.....

Reviewing cinder-scheduler on node-2 reveals that it indeed does not listen to 'cinder-scheduler', but it still listens for two queues:
 * cinder-scheduler.node-2.test.domain.local
 * cinder-scheduler_fanout_2034ead91d5c4d7584b353da65d6a9a2

Probably it is the same story for other queues.

Job where problem has reproduced: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/8.0.custom_system_test/1895/

[Dmitry Mescheryakov (dmitrymex)]

Snapshot from the job #1895

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Dmitry Mescheryakov (dmitrymex)]

We have never seen the issue again, probably it has gone away after last RabbitMQ upgrade. Invalidating it for now.