notification queues are created in rabbit but never consumed

Bug #1188643 reported by Michael Kerrin
74
This bug affects 11 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Medium
Unassigned
OpenStack Identity (keystone)
Invalid
Medium
Unassigned
devstack
Invalid
Undecided
Unassigned
oslo.messaging
Won't Fix
Undecided
Unassigned

Bug Description

The following queues are created in rabbit but there are no consumers for them. notifications.info, notifications.warn and notifications.error. This means that all events are queued up in them until rabbit is restarted or else someone consumes the queue.

notifications.info in particular collects a large number of events very quickly

All events should be published to an exchange and it should be up the consumers on how to configure any queues in rabbit and how they should be consumed.

Tags: ops
Revision history for this message
Russell Bryant (russellb) wrote :

Good point, in the case of notifications, we don't want to create the queues. The fix is actually in oslo, so i'm going to add it as an affected project.

Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Tiantian Gao (gtt116) wrote :

BTW, I found these is a Note in nova/openstack/common/rpc/impl_kombu.py:
*********
# NOTE(jerdfelt): Normally the consumer would create the queue, but
# we do this to ensure that messages don't get dropped if the
# consumer is started after we do
*********
The note also make sense...

Ben Nemec (bnemec)
Changed in oslo:
status: New → Confirmed
Revision history for this message
Dean Troyer (dtroyer) wrote :

Sounds like this isn't a DevStack issue. Reopen if needed.

Changed in devstack:
status: New → Incomplete
Revision history for this message
Jian Wen (wenjianhn) wrote :

I agree with Tiantian.
You may create a daemon to consume the queue as a work round.

Jian Wen (wenjianhn)
Changed in nova:
assignee: nobody → Jian Wen (wenjianhn)
Revision history for this message
mohsiddiquebagwan (mohdsiddiquebagwan) wrote :

Hi Jian,

I am also facing the same issue.
Could you please elaborate What do we need to do on controller side so that this error message get shown on horizon

Thanks
Mohd Siddique Bagwan

affects: oslo-incubator → oslo.messaging
Dean Troyer (dtroyer)
Changed in devstack:
status: Incomplete → Invalid
Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote :

Ceilometer will consume notifications.info for now, but yes, Ceilometer is not required for every environment, so rabbitmq can crash for a very long queue in such situation.

See bug: https://bugs.launchpad.net/ceilometer/+bug/1364708

Revision history for this message
Haneef Ali (haneef) wrote :

IMO cieliometer should not consume from this queue. If some other service start using this queue, then only one is going to get the message. Cieliometer should create a random queue and associate with the exchange so that it has exclusive access to the message

Revision history for this message
Haneef Ali (haneef) wrote :

I beleive just having an config option to create/don't create this queue should help.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Keystone is waiting on oslo.messaging fixes - depending on those changes keystone will make appropriate changes to enable the new "don't create a queue" or whatever the functionality ends up being.

Changed in keystone:
status: New → Confirmed
importance: Undecided → Medium
tags: added: ops
Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

IIRC, we create the queue from the publisher side at the same time that we define the topic so that if the publisher and subscriber are started in the "wrong" order the subscriber won't miss any of the messages. If there is never going to be anything listening to the queues, it should be possible to disable the publisher entirely by configuring the "noop" driver instead of the "messaging" or "messagingv2" driver should eliminate the problem.

Changed in nova:
status: Confirmed → Won't Fix
Changed in oslo.messaging:
status: Confirmed → Won't Fix
Jian Wen (wenjianhn)
Changed in nova:
assignee: Jian Wen (wenjianhn) → nobody
Changed in keystone:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.