[SRU] rabbit queues should expire when unused

[John Eckersberg]

[Description]

RabbitMQ supports queue-level TTLs as described here:

https://www.rabbitmq.com/ttl.html#queue-ttl

This should be used when declaring queues in order to clean up queues that are orphaned for various reasons.

This is an important complement to auto_delete queues. Queues marked auto_delete are only deleted if a consumer existed at some point, and then disconnected, resulting in zero consumers.

[Impact]

Consider the following scenario: a client declares a queue (auto_delete) and binds it to a fan out exchange, but before the client can consume from the queue, it dies.

Because there was never (and will never be) a consumer of this queue, the auto_delete logic does not fire. The queue will live forever, and will collect a copy of every message that is sent to the bound fanout exchange. Given enough published messages to the exchange, the queue will eventually consume all available memory on the broker.

This is bad and we should avoid it by setting a reasonable TTL.

[Test Case]

* Deploy a new cinder service unit
* Create a volume
* Destroy the new cinder service unit.
* List the current active queues:

rabbitmqctl -p openstack list_queues messages consumers name

811 0 q-agent-notifier-security_group update_fanout_c3c4f4d4fc774322938357952ab6d252
3352 0 cinder-volume_fanout_49a14d5511dc4c8c935904eb299a06a8
3352 0 cinder-volume_fanout_634a3e584c064a788676fb29fbf7db23
3352 0 cinder-volume_fanout_ada4f2368cb74d7389998d0adbd15f88
3352 0 cinder-volume_fanout_e896e25b029c4cbd97037909c34803db

Those queues with 0 consumers will grow and remain there forever.

After applying the patch, those queues are removed from the exchange after
10 minutes.

[Regression Potential]

* Not identified.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/243845
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=10625eed87b4c7f980bd5cd7cacbc4caa2dec197
Submitter: Jenkins
Branch: master

commit 10625eed87b4c7f980bd5cd7cacbc4caa2dec197
Author: John Eckersberg <email address hidden>
Date: Fri Nov 20 17:25:58 2015 -0500

    Kombu: make reply and fanout queues expire instead of auto-delete

    Right now fanout and reply queues are unconditionally created with
    auto-delete flag which causes a number of problems listed in bug
    1495568. Replacing auto-delete with queue expiration with some sane
    timeout should fix all these issues at once.

    Another problem being fixed is that auto-delete flag does not causes
    the queue to be deleted if it never had consumers. An orphaned fanout
    queue might appear that way and it will grow indefinitely until
    somebody manually removes it. See bug 1515278 for details.

    A new rabbit_transient_queues_ttl config parameter is introduced which
    configures the TTL for reply and fanout queues. It is a positive
    integer representing timeout in seconds. By default it is set to 10
    minutes. That should be enough for application to reconnect or
    for server to send reply to client which already died. At the same
    time, it seems that not so many messages could be accumulated in
    fanout queues during that time.

    DocImpact
    With this change RabbitMQ driver defines reply and fanout queues
    differently comparing with the previous release: now they are defined
    with queue TTL (https://www.rabbitmq.com/ttl.html#queue-ttl) instead
    of auto-delete flag. That helps avoid a number of issues, see commit
    description for details. A new rabbit_transient_queues_ttl parameter
    is defined which controls the TTL value. It is set to 10 minutes by
    default. The change does not affect upgrade in any way.

    Closes-bug: #1495568
    Closes-bug: #1515278

    Co-Authored-by: Dmitry Mescheryakov <email address hidden>
    Change-Id: I83a8d09dc0cdae24c12d7043ec810529a9ce57ab

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Trusty-Kilo Patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/279561

[Jorge Niedbalski (niedbalski)]

Dear Ubuntu Maintainers,

I attached a patch for backporting this fix in Ubuntu Trusty and Wily (Kilo, Liberty).

- https://bugs.launchpad.net/oslo.messaging/+bug/1515278/+attachment/4570118/+files/fix-lp-1515278-liberty.patch
- https://bugs.launchpad.net/oslo.messaging/+bug/1515278/+attachment/4569507/+files/fix-lp-1515278-kilo.patch

Also a proposal for backporting this into OpenStack Stable Liberty:

- https://review.openstack.org/279561

Thanks for your consideration.

[Iain Lane (laney)]

Jorge,

For an update to an Ubuntu stable release, please provide the required information as outlined in <https://wiki.ubuntu.com/StableReleaseUpdates#Procedure>.

Thanks!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/279561
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=393370072e9f8122967c4864c9dc8a7bda731f3a
Submitter: Jenkins
Branch: stable/liberty

commit 393370072e9f8122967c4864c9dc8a7bda731f3a
Author: Jorge Niedbalski <email address hidden>
Date: Fri Feb 12 11:20:46 2016 -0300

    Kombu: make reply and fanout queues expire instead of auto-delete

    Backport of fix made for bug #1515278.

    Right now fanout and reply queues are unconditionally created with
    auto-delete flag which causes a number of problems listed in bug
    1495568. Replacing auto-delete with queue expiration with some sane
    timeout should fix all these issues at once.

    Original: https://review.openstack.org/#/c/243845/

    Change-Id: Ib69fb7a159eab722fe399f64caa742b77ef45bec
    Closes-bug: #1515278
    Signed-off-by: Jorge Niedbalski <email address hidden>

[Corey Bryant (corey.bryant)]

Jorge,

I've uploaded this to wily and it will require SRU team review before it gets into wily-proposed. Can you update the bug description with the standard SRU sections? [Impact], [Test Case], and [Regression Potential].

Thanks,
Corey

[Chris J Arges (arges)]

Hello John, or anyone else affected,

Accepted python-oslo.messaging into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.messaging/2.5.0-1ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello John, or anyone else affected,

Accepted python-oslo.messaging into liberty-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:liberty-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-liberty-needed to verification-liberty-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-liberty-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ryan Beisner (1chb1n)]

@niedbalski FYI, tempest results from wily-liberty-proposed and trusty-liberty staging are consistent with wily-liberty (distro) and trusty-liberty (uca updates). ie. No new failures. Now that it has been promoted to trusty-liberty-proposed, we will need to re-test there. There is typically a 1-wk bake period for proposed cloud archive pockets.

[Corey Bryant (corey.bryant)]

Hi Chris and James,

python-oslo.messaging 2.5.0-1ubuntu2 has been verified on wily-proposed and trusty-liberty-proposed.

Thanks,
Corey

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.messaging - 2.5.0-1ubuntu2

---------------
python-oslo.messaging (2.5.0-1ubuntu2) wily; urgency=medium

  [ Jorge Niedbalski ]
  * d/p/make-reply-and-fanout-queues-expire-instead-of-auto-delete.patch:
    Make reply and fanout queues expire instead of auto-delete (LP: #1515278).

  [ Corey Bryant ]
  * d/p/dont-hold-connection-when-reply-fail.patch: Cherry-picked
    patch from upstream VCS to fix the amqp reply logic when
    connections are lost (LP: #1521958).

 -- Corey Bryant <email address hidden> Mon, 21 Mar 2016 08:08:47 -0400

[Chris J Arges (arges)]

The verification of the Stable Release Update for python-oslo.messaging has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

The verification of the Stable Release Update for python-oslo.messaging has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package python-oslo.messaging - 2.5.0-1ubuntu2~cloud0
---------------

 python-oslo.messaging (2.5.0-1ubuntu2~cloud0) trusty-liberty; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-oslo.messaging (2.5.0-1ubuntu2) wily; urgency=medium
 .
   [ Jorge Niedbalski ]
   * d/p/make-reply-and-fanout-queues-expire-instead-of-auto-delete.patch:
     Make reply and fanout queues expire instead of auto-delete (LP: #1515278).
 .
   [ Corey Bryant ]
   * d/p/dont-hold-connection-when-reply-fail.patch: Cherry-picked
     patch from upstream VCS to fix the amqp reply logic when
     connections are lost (LP: #1521958).

[Mathew Hodson (mhodson)]

The package is called oslo.messaging in the Ubuntu Trusty archive, so I added that task.

https://launchpad.net/ubuntu/trusty/+source/oslo.messaging

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.