Notifications queues filling up when deployed without ceilometer

[Sandor Zeestraten]

# Versions
rabbitmq-server, rev 61

We have a few openstack deployments without ceilometer where there are rabbitmq queues (notifications.error, notifications.info, notifications_designate.info) that are endlessly filling up. I'm guessing this is because there is no ceilometer instance to consume these queues.

As far as I can tell these are some of the ways to work around this, either set TTL's or max queue sizes on the rabbitmq queues, or tell the different openstack services to not report to rabbitmq by changing the notification_driver to noop.

All in all, this makes for a not so fun default behaviour as it also impacts the default nrpe rabbitmq_queue checks.

# Our current workaround
Set the message TTL to 10 min for all queues with name starting with notifications for the rabbitmq vhost called openstack (default).
  rabbitmqctl set_policy TTL "notifications*" '{"message-ttl":600000}' --priority 1 --apply-to queues -p openstack

Purge the messages already in the queue as they won't be cleared by the new TTL
  rabbitmqctl purge_queue notifications.error -p openstack
  rabbitmqctl purge_queue notifications.info -p openstack
  rabbitmqctl purge_queue notifications_designate.info -p openstack

[James Page (james-page)]

I think setting the TTL policy is an OK workaround; note that this type of reconfiguration in the event that ceilometer is or is not deployed would be covered by the service discovery spec (which although specified remains unimplemented to date).

[0] http://specs.openstack.org/charm-specs/specs/queens/approved/service-discovery.html

[Ryan Beisner (1chb1n)]

I think it's a reasonable feature request of the rmq charm, to allow users to use charm config and/or charm actions to manage operations of queues, including TTLs and purges.

But that would need to be proposed and done generically, so that it is a multi-purpose ops routine/config.

The rabbitmq-server charm is not just for openstack, and I'd say this won't be the first or last queue to fill up. ;-)

Having said my support for that, as a useful and generic feature for the charm, I disagree with that approach to address the underlying bug where: message topics are created and messages are collected on those topics, but no one is actually listening to them.

IMHO, those topics should not be created in the first place, and messages should not be posted.

[Junien F (axino)]

Filed bug 1761405 upstream.

[Junien F (axino)]

Note that if you want the TTL to apply to versioned notifications as well (in the workaround in the OP), you should use "^(versioned_)?notifications.*" as regexp.

[Junien F (axino)]

In bug 1761405 there's a workaround to avoid notifications being sent in the first place. Would be trivial-ish to add as a config option, at least in the 18.02 charms.

[Ryan Beisner (1chb1n)]

https://review.openstack.org/#/c/583580

[Paul Henien (phenien)]

This bug is also affecting the event.sample, alarm.all.sample and metering.sample queues.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583580
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=77b6175c1cfccd5cb78ef2de14bf70c4283b5cbc
Submitter: Zuul
Branch: master

commit 77b6175c1cfccd5cb78ef2de14bf70c4283b5cbc
Author: James Page <email address hidden>
Date: Wed Jul 18 09:47:49 2018 -0400

    Workaround notification topic filling with OpenStack

    OpenStack charms currently unconditionally enable notifications
    for consumption by aodh/ceilometer etc... If these services are
    not deployed, then notification queues will fill over time and
    cause alerts/service issues.

    Workaround this issue for now by setting a TTL on notification
    topics under the openstack vhost (if configured).

    By default, the TTL for messages in any notification topics will
    be set to 1 hour, after which they are dropped from the queue.

    Change-Id: I46a11e98a2e3bbb6016e7ddfe27bdffe55c37aab
    Partial-Bug: 1737170