Number of Rabbitmq queues is growing from failover to failover

[Alexey Khivin]

When using oslo.messaging with RabbitMQ driver

On every start Heat creates 4 queues with the names like

b21828bd-80c3-4abf-8f50-0ffd71671d6b
b21828bd-80c3-4abf-8f50-0ffd71671d6b.node-4
engine
engine.node-4

two of them have contant names and other two have random names.
So, the number of randomly named queues increases every time the Heat restarts

These queues have no "auto_delete" and "x-expire" options, so they will not be deleted until RabbitMQ cluster restarted

[Doug Hellmann (doug-hellmann)]

Is this the same as https://bugs.launchpad.net/oslo.messaging/+bug/1406629 ?

[Alexey Khivin (akhivin)]

Thanks for your reply Doug! #1406629 looks different

As i understand now, may be conf.amqp_auto_delete

mentioned
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L306
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L411

should be True for the Heat, but it seems to be False

[Alexey Khivin (akhivin)]

Let me explain in more details

Heat creates workers for some tasks. Each worker is a separate process.
And every workers consumes rpc calls via oslo.messaging.
It is possible to run several workers on the node in parallel.
So, to identify each worker it has your own uuid generated at the moment of worker creation.
This uuid used in queues names. After worker is dead, queue should be deleted but there is no way to specify auto_remove parameter for the oslo.messaging.Target or for the oslo.messaging.get_rpc_server

A litle more details

On the one hand if someone uses TopicConsumer

class TopicConsumer(ConsumerBase):
    """Consumer class for 'topic'."""

    def __init__(self, conf, channel, topic, callback, tag, exchange_name,
                 name=None, **kwargs):

or Publisher

class TopicConsumer(ConsumerBase):
    """Consumer class for 'topic'."""

    def __init__(self, conf, channel, topic, callback, tag, exchange_name,
                 name=None, **kwargs):

it is possible to specify some kwargs parameter for this particular queue/topic/fanout

On the ither hand if someone (Heat) uses rpc server like that

    def start(self):
        super(EngineListener, self).start()
        self.target = messaging.Target(
            server=self.host, topic=self.engine_id)
        server = rpc_messaging.get_rpc_server(self.target, self)
        server.start()

https://github.com/openstack/heat/blob/master/heat/engine/service.py#L232

There is no way to pass kwargs for the internal queue/topic/fanout

in the oslo.messaging/oslo_messaging/rpc/server.py

101 def get_rpc_server(transport, target, endpoints,
102 executor='blocking', serializer=None):

[Alexey Khivin (akhivin)]

It seems very straightforward, If we able to specify additional options when using TopicConsumer/TopicPublisher, then we should able to specify same options in case of using Target+get_rpc_server

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/159876

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159876
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9b27efc910695d1be92f2c293d5d11e35e9637f3
Submitter: Jenkins
Branch: master

commit 9b27efc910695d1be92f2c293d5d11e35e9637f3
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Feb 27 16:00:29 2015 +0100

    Change the engine-listener topic

    In oslo.messaging to identify a "rpc server" we have to set the
    server attribut of the target. But currently, the engine-listener uses
    the hostname of the host, that make all workers on the host to be
    identified has same "rpc server" by oslo.messaging, by chance because
    heat uses a different topics for each workers, that works.

    This change uses the engine id instead of the hostname to identify the
    "rpc server" and use the same topic for all engine listeners.

    Related-bug: #1414674

    Change-Id: I359709ab15ee24ac2459826264adf2373dbb067f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/161298

[Mehdi Abaakouk (sileht)]

oslo.messaging needs to be fixed too, by supporting x-expire on topic queue

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161298
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a052b06f9e88a1179ba72acc53c4212083025fdd
Submitter: Jenkins
Branch: stable/juno

commit a052b06f9e88a1179ba72acc53c4212083025fdd
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Feb 27 16:00:29 2015 +0100

    Change the engine-listener topic

    In oslo.messaging to identify a "rpc server" we have to set the
    server attribut of the target. But currently, the engine-listener uses
    the hostname of the host, that make all workers on the host to be
    identified has same "rpc server" by oslo.messaging, by chance because
    heat uses a different topics for each workers, that works.

    This change uses the engine id instead of the hostname to identify the
    "rpc server" and use the same topic for all engine listeners.

    Related-bug: #1414674

    Conflicts:
     heat/engine/service.py

    Change-Id: I359709ab15ee24ac2459826264adf2373dbb067f
    (cherry-picked from commit 9b27efc910695d1be92f2c293d5d11e35e9637f3)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179708

[Viktor Serhieiev (vsergeyev)]

Mehdi, are you plan to continue your work on this bug?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: master
Review: https://review.openstack.org/179708
Reason: Abandon this for now, I will perhaps repropose it later.

[Davanum Srinivas (DIMS) (dims-v)]

Fixed in https://review.openstack.org/#/c/243845/

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

Not sure this is really fixed:
- Launch a mitaka devstack
- reattach the screen (sreen -r)
- sudo rabbitmqctl list_queues | grep -i heat | wc -l # I get 33
- go to the h-eng session of screen, <ctrl-c> it and relaunch it.
- go to the "shell" session of screen and relaunch "sudo rabbitmqctl list_queues | grep -i heat | wc -l" I now have 65
- repeat again, I get 97 heat queues...

every time I stop/start h-eng :
/usr/local/bin/heat-engine --config-file=/etc/heat/heat.conf & echo $! >/opt/stack/status/stack/h-eng.pid; fg || echo "h-eng failed to start" | tee "/opt/stack/status/stack/h-eng.failure"
The heat queues increase.

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

Bug is not closed : https://bugs.launchpad.net/heat/+bug/1599104

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/357045