durability of rabbit exchange/queue should be configurable

Bug #1054183 reported by Eoghan Glynn on 2012-09-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
High
Eoghan Glynn

Bug Description

The durability of the rabbit exchange and queue is hard-coded to True.

Yet for other openstack services using the common RPC code, this is configurable and defaults to False.

If there's a mismatch between the consumer and producer on the durable flag, this leads to a failure like:

  AMQPChannelException: (406, "PRECONDITION_FAILED - cannot redeclare exchange 'glance' in vhost with different type, durable, internal or autodelete value, (40, 10), 'Channel.exchange_declare'")

The consumption of glance notifications by ceilometer would be less fragile if glance durability could be similarly configured.

Eoghan Glynn (eglynn) on 2012-09-21
Changed in glance:
assignee: nobody → Eoghan Glynn (eglynn)
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.openstack.org/13474

Changed in glance:
status: New → In Progress
Eoghan Glynn (eglynn) on 2012-09-21
Changed in glance:
milestone: none → folsom-rc2

Reviewed: https://review.openstack.org/13474
Committed: http://github.com/openstack/glance/commit/eafed25ba64f5fd8af97bcbe8e9f12a1fe4f7c9f
Submitter: Jenkins
Branch: master

commit eafed25ba64f5fd8af97bcbe8e9f12a1fe4f7c9f
Author: Eoghan Glynn <email address hidden>
Date: Fri Sep 21 15:32:49 2012 +0000

    Add rabbit_durable_queues config option.

    Fixes bug 1054183

    Avoid AMQPChannelException: (406, "PRECONDITION_FAILED...") failures
    due to a mismatch between the durability of rabbitmq exchange/queue
    declared by glance and ceilometer.

    Change-Id: I4e25986a1f503782e701aa1168c4eb231ff25d06

Changed in glance:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/13475
Committed: http://github.com/openstack/glance/commit/5268d6b1a71d1132a05cf989c322f44ec4445fb5
Submitter: Jenkins
Branch: milestone-proposed

commit 5268d6b1a71d1132a05cf989c322f44ec4445fb5
Author: Eoghan Glynn <email address hidden>
Date: Fri Sep 21 15:32:49 2012 +0000

    Add rabbit_durable_queues config option.

    Fixes bug 1054183

    Avoid AMQPChannelException: (406, "PRECONDITION_FAILED...") failures
    due to a mismatch between the durability of rabbitmq exchange/queue
    declared by glance and ceilometer.

    Change-Id: I4e25986a1f503782e701aa1168c4eb231ff25d06

Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-09-27
Changed in glance:
milestone: folsom-rc2 → 2012.2
Jay Pipes (jaypipes) wrote :

This has not been fixed...

On a fresh checkout of devstack, I now run into a glance service that won't respond. This, however, isn't the worst part. The worst part is that the glance API server is hung in a perpetual loop, eating up disk space for logs at an alarming rate.

After just a few seconds, the glance API log file (screen-ga-api.log) was at 17MB:

jpipes@uberbox:~/repos/devstack$ ls -alh /opt/stack/logs/screen-g-api*
-rw-rw-r-- 1 jpipes jpipes 17M Nov 1 13:56 /opt/stack/logs/screen-g-api.2012-11-01-134816.log
lrwxrwxrwx 1 jpipes jpipes 50 Nov 1 13:49 /opt/stack/logs/screen-g-api.log -> /opt/stack/logs/screen-g-api.2012-11-01-134816.log

The log file looks like this: http://paste.openstack.org/show/23512/

with the child process attempting to create the consumer, failing, and then the parent process launching another child process, which does the same thing, continuing ad-infinitum.

Basically, at a minimum, I believe that the AMQP negotiation should at least be *tried* before the child processes are spawned so that basic things like this can be caught before the child process starts. Either that, or the parent must have a wait to stop spawning on repeated failures.

Brian Waldon (bcwaldon) wrote :

We can definitely reproduce the mismatch of durability between the server and the client, but this bug covers the ability to configure the durability itself. I filed a new bug to track the issues (which I can replicate on my machine) that you're running into, Jay : https://bugs.launchpad.net/glance/+bug/1074132

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers