guestagent queue is not deleted when trove instance is deleted

[WonChon]

When trove instance was deleted, guestagent queue is not deleted.
So, rabbitmq has too many unused queues.
Sometimes, some queues for deleted instances has orphan messages.

[Amrith Kumar (amrith)]

Under what circumstances, also please dump these messages which are left in the queues.

[WonChon (arisu1000)]

Remained message is like following:

{"oslo.message": "{\"_context_domain\": null, \"_msg_id\": \"f0e9b796886148659ded18bc531e8290\", \"_context_read_only\": false, \"_context_request_id\": \"req-9323674a-1c19-428e-bfaf-cd18980bf070\", \"_context_service_catalog\": [{\"endpoints\": [], \"type\": \"database\", \"name\": \"trove\"}, {\"endpoints\": [], \"type\": \"identity\", \"name\": \"keystone\"}], \"args\": {\"fs_path\": null}, \"_unique_id\": \"38225976d6cc401db18a81047b29d486\", \"_context_instance_uuid\": null, \"_context_marker\": null, \"_context_auth_token\": \"1ade9078303a48be923c65c5e46ddcf2\", \"_context_user_identity\": \"8c97b8bc5632452ea8d6e92b1c6eed66 e9404cc1fbaf4f229552d9222e17bb07 - - -\", \"_reply_q\": \"reply_7c0c349792f24f6a973352ba09c24da1\", \"_context_show_deleted\": false, \"_context_tenant\": \"e9404cc1fbaf4f229552d9222e17bb07\", \"_context_is_admin\": false, \"version\": \"1.0\", \"_context_project_domain\": null, \"_context_user\": \"8c97b8bc5632452ea8d6e92b1c6eed66\", \"_context_user_domain\": null, \"method\": \"get_filesystem_stats\", \"_context_limit\": null}", "oslo.version": "2.0"}

[WonChon (arisu1000)]

To reproduce this issue I tried following:

1. Delete trove instance in horizon.
2. Reload trove instance list page in horizon when deleting trove instance.

I suspect that trove taskmanager request file system info to guestagent but, nova instance already have been deleted. So, the queue of this trove instance can not consume.

[Matt Fischer (mfisch)]

I can reproduce as well using Tesora trove 1.9.1 which is Mitaka++ I believe. I had 18 messages in the queue.

[Doug Shelley (0-doug)]

As far as I can tell, there isn't a way to just delete a topic queue via oslo.messaging. The options appear to be:
1. Set a global policy to expire queues (can be pinned to "guestagent.*") when all consumers are disconnected.
2. Switch from using "topic" queues to "direct" queues which have expiries set. I'm not sure exactly how to do this but it would certainly require a change to Trove.
3. Switch to using the Pika driver for rabbit (i.e. transport_url=pika://...). Pika has a setting for queue expiration. This is a newer oslo.messaging driver; not sure how comfortable folks would be to switch out rabbit/kombu for this.

Note that all the options basically come back to having queues automatically deleted by the broker after some period of non-activity (i.e. no consumers). This appears to be the only "sane" way to clean up queues.

One thing to keep in mind - if a guest gets disconnected from the broker for some temporary reason (e.g. process failure or network partition) but is away long enough that it's queue get deleted, the queue will be recreated reliably on reconnect. However, I've found a couple of cases where the recreated queue isn't useabe by API/TaskMgr.

I will be filing separate bugs on these and will put references in here.

[Doug Shelley (0-doug)]

The 2 related bugs mentioned above:
https://bugs.launchpad.net/trove/+bug/1627170
https://bugs.launchpad.net/trove/+bug/1627179