[messaging] qpid driver leaks queues and exchanges

[Ken Giusti]

When using the QPID transport, it seems that RPC client activity creates queues and exchanges on the broker for receiving replies from the RPC server. However, these temporary queues/exchanges are not deleted once the client exits. Over time, this will unnecessarily impact the broker's resources.

Here's an example, I've created a test RPC server that is listening for requests to "my-topic". Using the qpid-stat tool, I'll dump the currently active exchanges and queues on the server:

(pyenv)[kgiusti@t530 messaging (new-addressing)]$ qpid-stat -e
Exchanges
  exchange type dur bind msgIn msgOut msgDrop byteIn byteOut byteDrop
  ===========================================================================================
                      direct 4 0 0 0 0 0 0
  amq.direct direct Y 0 0 0 0 0 0 0
  amq.fanout fanout Y 0 0 0 0 0 0 0
  amq.match headers Y 0 0 0 0 0 0 0
  amq.topic topic Y 0 0 0 0 0 0 0
  my-topic_fanout fanout 1 0 0 0 0 0 0
  openstack topic Y 2 0 0 0 0 0 0
  qmf.default.direct direct 0 0 0 0 0 0 0
  qmf.default.topic topic 1 40 1 39 40.0k 6.29k 33.7k
  qpid.management topic 0 0 0 0 0 0 0

(pyenv)[kgiusti@t530 messaging (new-addressing)]$ qpid-stat -q
Queues
  queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind
  =======================================================================================
  039455f9-7bbe-40b7-8ce1-2c81c9427ec9:0.0 Y Y 0 0 0 0 0 0 1 2
  my-topic 0 0 0 0 0 0 1 2
  my-topic.SERVER1 0 0 0 0 0 0 1 2
  my-topic_fanout_82f50782f0be46f18bdb5571432e6402 Y Y 0 0 0 0 0 0 1 2

The server has created a couple of exchanges and queues, as expected. Now if I run a simple client that performs one RPC call against that server:

(pyenv)[kgiusti@t530 work (master)]$ ./my-client.py my-topic methodA arg1 arg2
Calling server on topic my-topic, server=None exchange=my-exchange namespace=my-namespace fanout=None
Method=methodA, args={'arg1': 'arg2'}
Return value=None

(pyenv)[kgiusti@t530 work (master)]$ qpid-stat -q
Queues
  queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind
  ================================================================================================
  1134624d-a4c1-477d-a3bf-a02177059890:0.0 Y Y 0 0 0 0 0 0 1 2
  my-topic 0 1 1 0 393 393 1 2
  my-topic.SERVER1 0 0 0 0 0 0 1 2
  my-topic_fanout_82f50782f0be46f18bdb5571432e6402 Y Y 0 0 0 0 0 0 1 2
  reply_c6c3c9e3e3e94d58bf19a134acf2880c 0 2 2 0 362 362 0 2

(pyenv)[kgiusti@t530 work (master)]$ qpid-stat -e
Exchanges
  exchange type dur bind msgIn msgOut msgDrop byteIn byteOut byteDrop
  ===================================================================================================
                                          direct 5 2 2 0 362 362 0
  amq.direct direct Y 0 0 0 0 0 0 0
  amq.fanout fanout Y 0 0 0 0 0 0 0
  amq.match headers Y 0 0 0 0 0 0 0
  amq.topic topic Y 0 0 0 0 0 0 0
  my-topic_fanout fanout 1 0 0 0 0 0 0
  openstack topic Y 2 1 1 0 393 393 0
  qmf.default.direct direct 0 0 0 0 0 0 0
  qmf.default.topic topic 1 98 3 95 101k 21.6k 80.0k
  qpid.management topic 0 0 0 0 0 0 0
  reply_c6c3c9e3e3e94d58bf19a134acf2880c direct Y 1 0 0 0 0 0 0

The output from qpid-stat shows that a new exchange and queue have been created by the client, and remain after the client has exited. I suspect this is unintended - the names appear to be random UUIDs, which means they'll probably not be used again.

Rerunning the client a couple of more times:

(pyenv)[kgiusti@t530 work (master)]$ qpid-stat -q
Queues
  queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind
  =================================================================================================================================
  0af107d8-fbae-4d7e-b976-557d75b7e337:0.0 Y Y 0 0 0 0 0 0 1 2
  my-topic 0 4 4 0 1.57k 1.57k 1 2
  my-topic.SERVER1 0 0 0 0 0 0 1 2
  my-topic_fanout_82f50782f0be46f18bdb5571432e6402 Y Y 0 0 0 0 0 0 1 2
  reply_5e6a03a1738a4d09bde350141fed08b7 0 2 2 0 362 362 0 2
  reply_6b5885ed87f74fdd9921e08718bd45f7 0 2 2 0 362 362 0 2
  reply_c6c3c9e3e3e94d58bf19a134acf2880c 0 2 2 0 362 362 0 2
  reply_f1434efea6424986997eb9b30141c809 0 2 2 0 362 362 0 2
(pyenv)[kgiusti@t530 work (master)]$ qpid-stat -e
Exchanges
  exchange type dur bind msgIn msgOut msgDrop byteIn byteOut byteDrop
  ===============================================================================================================
                                          direct 8 8 8 0 1.45k 1.45k 0
  amq.direct direct Y 0 0 0 0 0 0 0
  amq.fanout fanout Y 0 0 0 0 0 0 0
  amq.match headers Y 0 0 0 0 0 0 0
  amq.topic topic Y 0 0 0 0 0 0 0
  my-topic_fanout fanout 1 0 0 0 0 0 0
  openstack topic Y 2 4 4 0 1.57k 1.57k 0
  qmf.default.direct direct 0 0 0 0 0 0 0
  qmf.default.topic topic 1 144 5 139 166k 42.3k 123k
  qpid.management topic 0 0 0 0 0 0 0
  reply_5e6a03a1738a4d09bde350141fed08b7 direct Y 1 0 0 0 0 0 0
  reply_6b5885ed87f74fdd9921e08718bd45f7 direct Y 1 0 0 0 0 0 0
  reply_c6c3c9e3e3e94d58bf19a134acf2880c direct Y 1 0 0 0 0 0 0
  reply_f1434efea6424986997eb9b30141c809 direct Y 1 0 0 0 0 0 0

Given that these queues & exchanges are not auto-delete, they'll have to be manually deleted.

[David Geng (genggjh)]

Will this issue effect the qpid performance?

In our production env, after about 6 hours, it will have above 5w queues and exchanges have be generated. The qpid-config will have no response after about 10 hours, I have to restart qpidd to resolve it.

  fffe613d84ff44709941379c5a526c07 direct Y 0 0 0 0 0 0 0
  fffe6c89785b498293b91435b392f63b direct Y 0 0 0 0 0 0 0
  fffffeb96d664ad2a399e3368d2567a7 direct Y 0 0 0 0 0 0 0
  heat topic Y 2 28 28 0 0 0 0
  network_fanout fanout 10 0 0 0 0 0 0
  nova topic Y 30 53.8k 53.8k 7 0 0 0
  openstack topic Y 5 24 23 1 0 0 0
  qmf.default.direct direct 0 1 1 0 0 0 0
  qmf.default.topic topic 1 360k 4.30k 355k 0 0 0
  qpid.management topic 0 358k 53 358k 0 0 0
  scheduler_fanout fanout 1 3.94k 3.94k 0 0 0 0
[root@os-controller-srv ~]# qpid-stat -e |wc -l
53778

[root@os-controller-srv ~]# qpid-config
Total Exchanges: 54184
          topic: 6
        headers: 1
         fanout: 10
         direct: 54167

   Total Queues: 66
        durable: 0
    non-durable: 66

[Ken Giusti (kgiusti)]

> Will this issue effect the qpid performance?

Yes - as the number of queues and exchanges builds up, qpidd will need more compute and memory resources to deal with them. Since - I suspect - these queues/exchanges are never re-used, then this wastes resources.

However, what you are experiencing seems to be different:

> [root@os-controller-srv ~]# qpid-config
> Total Exchanges: 54184
> topic: 6
> headers: 1
> fanout: 10
> direct: 54167
>
> Total Queues: 66
> durable: 0
> non-durable: 66

The bug I'm experiencing causes a large number of exchanges _and_ queues. The number of queues in the above output seems reasonable, where the number of direct exchanges seems excessive.

In my case, if I run

$ qpid-stat -e | grep 'reply_'

and

$ qpid-stat -q | grep 'reply_'

I see many queues and exchanges named with the prefix "reply_"

I suspect you are experiencing a different bug which also triggers an exchange leak.

[Mark McLoughlin (markmc)]

Thanks for the report Ken

Could you confirm that this is covered by the fix for bug #1178375 ?

[Ken Giusti (kgiusti)]

Hi Mark,

That fix does seem to prevent the extra exchanges from being created. But even with the fix, I'm seeing the number of queues increase by one with each client RPC request. Appears as if the client reply queues are still being created without auto-delete being configured.

I'll look at this a bit more on Monday - my test clients use oslo.messaging, not oslo-incubator, and I pulled the patch over manually - I could've botched it.

[Ken Giusti (kgiusti)]

Hi Mark,

Yes, I do believe that the fix for https://bugs.launchpad.net/bugs/1178375 does in fact fix this bug.

However, even with this patch I'm seeing the following behavior:

1) reply queues are still present after the RPC has completed. This is due to the default setting for the configuration option "amqp_auto_delete" is false:

    cfg.BoolOpt('amqp_auto_delete',
                default=False,
                help='Auto-delete queues in amqp.'),

So technically, this isn't a bug, but you may want to have temporary reply queues auto-deleted by default if they are created for each rpc call.

2) An rpc call on a topic is being received by all listeners instead of being load-balanced to one of the listeners. The addressing configuration for TopicConsumer will create a separate queue for each subscriber and deliver a copy to each queue. I don't believe this is the intended pattern - disregard this if I'm mistaken.

Regardless, this bug should be closed as a duplicate of 1178375. Let me know what you think of the other issues, and I can open a new bug(s)/supply patches as necessary.

-K