Environment:
Fuel 6.0 2014.2 based environment with fresh oslo.messaging 1.4.1 from fuel 6.1
3 controller nodes, 2 compute nodes, 3 ceph nodes
RabbitMQ went into a looping on one controller node as below,
tail -f /<email address hidden>
=ERROR REPORT==== 23-Jul-2015::07:41:47 ===
connection <0.14200.0>, channel 1 - soft error:
{amqp_error,not_found,
"no queue 'reply_2e48d0e4650e4de3a200022406c27dea' in vhost '/'",
'basic.consume'}
=ERROR REPORT==== 23-Jul-2015::07:41:48 ===
connection <0.14200.0>, channel 1 - soft error:
{amqp_error,not_found,
"no queue 'reply_2e48d0e4650e4de3a200022406c27dea' in vhost '/'",
'basic.consume'}
=ERROR REPORT==== 23-Jul-2015::07:41:49 ===
connection <0.14200.0>, channel 1 - soft error:
{amqp_error,not_found,
"no queue 'reply_2e48d0e4650e4de3a200022406c27dea' in vhost '/'",
'basic.consume'}
^C
root@node-2:/etc# rabbitmqctl list_queues reply_2e48d0e4650e4de3a200022406c27dea
Listing queues ...
Error: {bad_argument,reply_2e48d0e4650e4de3a200022406c27dea}
When I restart this node, the error switch to another controller node, once it switch to rabbitmq master node, the horizon service for create & delete instance will be unavailable.
I restored the service by command: crm resource restart p_rabbitmq-server
this bug seems to be a race condition that queue.declare is handled before mq delete the.queue.
and the client continous basic.consume on an non-exist queue is definitely raise an.error.
This bug has been fixed in 1.5.1 by using kombu implemented ensure.
Maybe you should use cli like this, rabbitmqctl list_queues |grep replyxxx