Fanout queues might grow pretty big before expire
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
oslo.messaging |
Confirmed
|
Undecided
|
Kirill Bespalov |
Bug Description
The issue is originally reported by Sam Morrison in thread http://
OpenStack version: Liberty
Steps to reproduce:
* Restart all Neutron agents in an OpenStack one by one.
* Notice that a number of fanout queues without consumers were left in rabbitmq. You can see them by running 'rabbitmqctl list_queues consumers messages name | sort -nr'
* The queues will live until they expire (time is controlled by rabbit_
That produces a pretty big load on RabbitMQ on big environment, here is some math from Sam's email:
"A bad scenario is when you make a change to your cloud that means all your 1000
neutron agents are restarted, this causes a couple of dead queues per agent to
hang around. (port updates and security group updates) We get around 25 messages
/ second on these queues and so you can see after 10 minutes we have a ton of
messages in these queues.
1000 x 2 x 25 x 600 = 30,000,000 messages in 10 minutes to be precise."
Aside from suggesting people to lower the expiration time to reduce impact, we can also help by deleting fanout queues on graceful shutdown. That will not help in case a service forcibly dies, but that covers pretty natural case when all agents are restarted after Neutron is updated.
description: | updated |
Changed in oslo.messaging: | |
assignee: | nobody → Kirill Bespalov (k-besplv) |
status: | New → In Progress |
Reviewed: https:/ /review. openstack. org/346732 /git.openstack. org/cgit/ openstack/ oslo.messaging/ commit/ ?id=a6f0aaed3ff 2e2ce40665d4e90 e92da9d3b3c753
Committed: https:/
Submitter: Jenkins
Branch: master
commit a6f0aaed3ff2e2c e40665d4e90e92d a9d3b3c753
Author: Kirill Bespalov <email address hidden>
Date: Mon Jul 25 15:11:53 2016 +0300
Delete fanout queues on gracefully shutdown
No reasons to kept fanout queues in case then
a rpc server is gracefully shutdown. The expiration
time of the fanout queue is too long (30 mins), so for
large scales it can accumulate a lot of messages before it be removed
Closes-Bug: 1606213 042f3a5424d70f8 7d486693024
Change-Id: Ieaa35c454df542