Exclusive queues cause reconnection errors

[Vish Ishaya]

From http://lists.openstack.org/pipermail/openstack-dev/2012-December/004184.html

I am seeing an issue with the FanoutConsumer: currently each nova service declares a FanoutConsumer (in addition to TopicConsumer and DirectConsumer). The FanoutConsumer declares a queue on the rabbitmq server, whose name is “<topic>_fanout_<UUID>”, for example: compute_fanout_e494cb57953b4e6d9dc68c72fb3e26b6.

This queue is marked as “exclusive” and “auto-delete”. “Exclusive” means that the queue can be used by a single Connection. However, when the FanoutConsumer detects disconnection and attempts to reconnect, it creates a new BrokerConnection, but uses the same queue name. In most cases, since the queue is marked as “auto-delete”, the rabbitmq server deletes the queue automatically on disconnection (after some timeout). So when FanoutConsumer tries to reconnect, the previous queue is gone. In some cases, however, the queue has not been deleted on the server yet, so when FanoutConsumer reconnects, there is an exception on the server:
=ERROR REPORT==== 26-Dec-2012::18:44:55 ===
connection <0.747.0>, channel 1 - error:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue 'volume_fanout_dcc7e1a2d71845d29cd53cfab8920289' in vhost '/'",
'queue.declare'}

As a result, the call to queue.declare() that leads to exception on the server is stuck and never returns. So the service is not able to receive any messages.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/18735

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/18735
Committed: http://github.com/openstack/oslo-incubator/commit/c51e68136903bbfa8275185a1656f6f5a3ebd2b6
Submitter: Jenkins
Branch: master

commit c51e68136903bbfa8275185a1656f6f5a3ebd2b6
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Dec 28 15:14:50 2012 -0800

    Don't use exclusive queues for fanout and direct

    The use of exclusive queues can cause issues on reconnect if the
    rabbit server hasn't deleted the queue already. We might as well
    use exclusive=False so that we can reconnect to the queue if
    necessary.

    Fixes bug 1094358

    Change-Id: I35bd903737d6054e8856b0003b0b8af97fd21c91