Number of heat queues will keep growing forever after restart heat-engine
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat Charm |
Fix Released
|
Undecided
|
Hua Zhang | ||
OpenStack RabbitMQ Server Charm |
Fix Released
|
Undecided
|
Hua Zhang |
Bug Description
Both lp:1599104 [1] and lp:1414674 [2] don't fix the problem.
and the problem can be easily reproduced according to their bug description or one comment [3].
- Launch a heat test env
- Get queue number before the test
# rabbitmqctl list_queues -p openstack | grep -E 'engine_
100
- restart heat-engine to trigger the problem
juju ssh heat/0 -- sudo systemctl restart heat-engine
- queue number will increase to 108
rabbitmqctl list_queues -p openstack | grep -E 'engine_
108
Then the queue number will reduce to 104
rabbitmqctl list_queues -p openstack | grep -E 'engine_
104
That's because there are 4 fanout queues, so they disappeared due to rabbit_
while the other 4 topic queues without TTL setting will be there forever.
rabbitmqctl list_queues -p openstack | grep -E 'engine_
engine_
engine_
heat-engine-
heat-engine-
- and so it's going to produce more topic queues without TTL setting until the vicious circle.
Why does this problem happen? That's because heat is using random UUID as the queue name, and that's consistent with the design. Because heat creates workers for some tasks. Each worker is a separate process. And every worker consumes rpc calls via oslo.messaging. It is possible to run several workers on the node in parallel. So, to identify each worker it has your own uuid generated at the moment of worker creation. This uuid used in queues names.
So I think the best solution is as this patch [6] does, also modify charm to support the following TTL setting.
rabbitmqctl set_policy heat_expiry "heat-engine-
[1] https:/
[2] https:/
[3] https:/
[4] https:/
[5] https:/
[6] https:/
tags: | added: sts |
Changed in charm-heat: | |
assignee: | nobody → Hua Zhang (zhhuabj) |
Changed in charm-heat: | |
status: | New → Invalid |
Changed in charm-heat: | |
milestone: | none → 21.10 |
Changed in charm-rabbitmq-server: | |
milestone: | none → 21.10 |
Changed in charm-heat: | |
status: | Fix Committed → Fix Released |
Changed in charm-rabbitmq-server: | |
status: | Fix Committed → Fix Released |
Fix proposed to branch: master /review. opendev. org/c/openstack /charm- rabbitmq- server/ +/787909
Review: https:/