Detailed bug description:
RabbitMQ OCF RA uses `rabbitmqctl list_queues`/`list_channels` as a health-check.
E.g. a 200-node OpenStack installation produces aronud 10k queues and
10k channels. Doing single list_queues/list_channels in cluster in this
environment results in 27k TCP packets and around 12 megabytes of
network traffic. Given that this calls happen ~10 times a minute with 3
controllers, it results in pretty significant overhead.
But as `list_queues`/`list_channels` are actually used as a health-check, it doesn't make sense
to check anything except queues/channels that exist on current node.
RabbitMQ 3.6.6 will contain some improvements that will allow us to perform such local-only checks:
- https://github.com/rabbitmq/rabbitmq-server/pull/883
- https://github.com/rabbitmq/rabbitmq-server/pull/911
- https://github.com/rabbitmq/rabbitmq-server/pull/915
Reproducibility:
100%
Impact:
When somebody else (e.g. LMA) starts to regularly call `list_queues` rabbit cluster can experience network split.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/mitaka
commit b9cb86a51fc1f60
Author: Alexey Lebedeff <email address hidden>
Date: Mon Aug 22 14:30:12 2016 +0300
Monitor rabbitmq from OCF with less overhead
This will stop wasting network bandwidth for monitoring.
E.g. a 200-node OpenStack installation produces aronud 10k queues and
10k channels. Doing single list_queues/
environment results in 27k TCP packets and around 12 megabytes of
network traffic. Given that this calls happen ~10 times a minute with 3
controllers, it results in pretty significant overhead.
Upstream change:
- https:/
To enable those features you shoud have rabbitmq containing following patches:
- https:/
- https:/
- https:/
Change-Id: Icfde3360b42a84
Closes-Bug: 1614071