Number of heat queues will keep growing forever after restart heat-engine

[Hua Zhang]

Both lp:1599104 [1] and lp:1414674 [2] don't fix the problem.

and the problem can be easily reproduced according to their bug description or one comment [3].

- Launch a heat test env

- Get queue number before the test

# rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |wc -l
100

- restart heat-engine to trigger the problem

juju ssh heat/0 -- sudo systemctl restart heat-engine

- queue number will increase to 108

rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |wc -l
108

Then the queue number will reduce to 104

rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |wc -l
104

That's because there are 4 fanout queues, so they disappeared due to rabbit_transient_queues_ttl=600 [4].

while the other 4 topic queues without TTL setting will be there forever.

rabbitmqctl list_queues -p openstack | grep -E 'engine_worker|heat-engine-listener' |grep -E '42cee820-4f0c-4aef-b8b6-705e7db3253a|8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510'
engine_worker.42cee820-4f0c-4aef-b8b6-705e7db3253a 0
engine_worker.8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510 0
heat-engine-listener.42cee820-4f0c-4aef-b8b6-705e7db3253a 0
heat-engine-listener.8c12d70c-b00c-4e9b-b33e-7cbf0cb8c510 0

- and so it's going to produce more topic queues without TTL setting until the vicious circle.

Why does this problem happen? That's because heat is using random UUID as the queue name, and that's consistent with the design. Because heat creates workers for some tasks. Each worker is a separate process. And every worker consumes rpc calls via oslo.messaging. It is possible to run several workers on the node in parallel. So, to identify each worker it has your own uuid generated at the moment of worker creation. This uuid used in queues names.

So I think the best solution is as this patch [6] does, also modify charm to support the following TTL setting.

rabbitmqctl set_policy heat_expiry "heat-engine-listener|engine_worker" "{'expires':3600000}" -p openstack --apply-to queues --priority 1

[1] https://bugs.launchpad.net/heat/+bug/1599104
[2] https://bugs.launchpad.net/heat/+bug/1414674
[3] https://bugs.launchpad.net/heat/+bug/1599104/comments/25
[4] https://review.opendev.org/c/openstack/oslo.messaging/+/243845/5/oslo_messaging/_drivers/impl_rabbit.py#170
[5] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1164
[6] https://review.opendev.org/c/openstack/fuel-library/+/356272/3/files/fuel-ha-utils/policy/set_rabbitmq_policy

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/787909

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-heat/+/792795

[Hua Zhang (zhhuabj)]

According to gnuoy's comment for patchset 3 in the page [1], I've refactored the code and implemented his idea. so now we have two patches:

1, heat side, see [2]
2, rabbitmq-server side, still see [1]

[1] https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/787909
[2] https://review.opendev.org/c/openstack/charm-heat/+/792795

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/787909
Committed: https://opendev.org/openstack/charm-rabbitmq-server/commit/707fa0e093af3b6ab636b65b018c29049f7a51ff
Submitter: "Zuul (22348)"
Branch: master

commit 707fa0e093af3b6ab636b65b018c29049f7a51ff
Author: Zhang Hua <email address hidden>
Date: Thu May 20 19:50:31 2021 +0800

    Number of heat queues will keep growing forever after heat-engine restarts

    Set TTL as a solution for topic queue engine_worker and heat-engine-listener
    to avoid them growing all the time after heat-engin restarts.

    This is rabbitmq-server part. eg: we can set heat ttl by:

    juju config heat ttl=3600000

    Closes-Bug: 1925436
    Change-Id: I7b826fe965a200da29020a8f2c6148f76d10a2b0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-heat/+/792795
Committed: https://opendev.org/openstack/charm-heat/commit/de88ad5344c4411dd81c600f724b760a0e8cf03f
Submitter: "Zuul (22348)"
Branch: master

commit de88ad5344c4411dd81c600f724b760a0e8cf03f
Author: Zhang Hua <email address hidden>
Date: Thu May 20 20:06:13 2021 +0800

    Number of heat queues will keep growing forever after heat-engine restarts

    Set TTL as a solution for topic queue engine_worker and heat-engine-listener
    to avoid them growing all the time after heat-engin restarts.

    This is heat part.

    Closes-Bug: 1925436
    Change-Id: I196346e4ca869efab45d1c2aafb1420b2a917d39

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-heat/+/798329

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/798410

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Zhang Hua <email address hidden>" on branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-heat/+/798329
Reason: The charm-rabbitmq-server fix 453b8e9 is not in stable/21.04, so this CI can't pass, the custome has applied the workaround and the code has already been in master, let's abandon it now.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Zhang Hua <email address hidden>" on branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-rabbitmq-server/+/798410
Reason: The charm-rabbitmq-server fix 453b8e9 is not in stable/21.04, so this CI can't pass, the custome has applied the workaround and the code has already been in master, let's abandon it now.