metering.sample queue grows with no consumer

[David Britton]

openstack: ocata
distro: 16.04

root@juju-a7951b-1-lxd-2:~# rabbitmqctl list_queues -p openstack |grep sample
alarm.all.sample 20
event.sample 20
metering.sample 426
notifications.sample 0

This is a non-HA 6 node deployment. on an HA deployment up for a few days, this number was already over 6000 growing at the rate of about 1k per day.

These messages should either be consumed, or should go in with a TTL.

[Fairbanks. (fairbanks)]

Hello there,

I'm exeperincing the same problems.
With a lot of activity like creating and destroying instances these ceilometer queue are getting very large like over 45000+ items in just a day.

I currently have ceilometer disabled now at one environment to stop these messages.
But i can't remove ceilometer at an other, it would be nice if it is possible to mitigate this some how.

Also, since i do not see the usage tab any more in the horizon dashboard, it doesn't make sense anyway of enabling this. But i think that is more a charm-openstack-dashboard problem?

[James Page (james-page)]

I'm having trouble re-producing this problem with the latest development charms - I'm running tempest against the deployed cloud and I see messages going through these queues, but they are consumed by ceilometer and aodh correctly afaict.

I'll try to reproduce with stable charms, but bearing in mind we're going to release this week that effort might not be worth it; I've done quite a bit of work on the telemetry stack this cycle so this may have been swept up in one of a number of other changes to ceilometer and notification configuration across the charm set.

[James Page (james-page)]

If you are *not* running ceilometer I can see how these queues would build up - we have a spec planned for early next cycle which should help services auto-configure themselves correctly based on what other services are deployed in the cloud.

[James Page (james-page)]

(Scrub #3 that might not be correct)

[Gábor Mészáros (gabor.meszaros)]

Seems similar to designate issue: https://bugs.launchpad.net/charms/+source/neutron-api/+bug/1659943

[Gábor Mészáros (gabor.meszaros)]

for my use-case it turned out that ceilometer units were needed to be shutdown (all at once) and then purging of the queue succeeded. With running ceilometer the queue was ever increasing.

[Drew Freiberger (afreiberger)]

@james-page

I'm seeing this for alarm.all.sample and event.sample coming OUT of ceilometer and destined for aodh, but this cloud has no aodh related to ceilometer. Must we install AODH to consume these queues, or will the next batch of charm fixes determine whether ceilometer should be generating samples for alerting depending on aodh relation? We've only just begun seeing alerts on these queues in rabbitmq filling up on xenial/ocata 17.08 charms environments.

[James Page (james-page)]

Marking designate task as invalid as this bug covers ceilometer -> aodh notifications for events and alarms.

Short answer would be to deploy aodh to consume this information; however we should update ceilometer to only generate the notifications when aodh is actually deployed.

[Tilman Baumann (tilmanbaumann)]

As a workaround, the charm could configure a TTL policy for the queue. That way the queue would at least not grow indefinitely.
Or even better perhaps use max-length. https://www.rabbitmq.com/maxlength.html

rabbitmqctl set_policy ceilometer-overflow metering.sample '{"max-length":20000}' --apply-to queues

[Xav Paice (xavpaice)]

In another very similar case, we have event.sample also:

~$ sudo rabbitmqctl -p openstack list_queues name messages consumers state

event.sample 2129 0 running
notifications_designate.info 190 0 running

There's actually a number of instances where rabbit queues keep on growing, event.sample being one of the bigger ones, but also notifications_designate.* as well as metering.sample.

I'm guessing that this is the same bug?

[Edward Hope-Morley (hopem)]

hi @xavpaice I think i know why you see your event.sample queue piling up with messages. This appears to quite simply be down to the fact that we are still (in queens) configuring the event_sink to publish to notifier:// and notifier://?topic=aodh [2] which results in both the aodh and event (default) topics being used hence why messages end up in both. This is a problem for two reasons (1) if you don;t have aodh deployed the messages will pile up in queues listening on the aodh and ceilometer exchanges such as alarm.all.sample and event.sample. Upstream queens simple has this configured to gnocchi [1] and if ever Panko got charmed we could set it to that as well. So solution-wise i think we need to (a) only set to aodh if aodh is related and (b) default to gnocchi if not aodh and (c) remove all publishers if neither is available since ceilometer itself will not process these events anymore.

[1] https://github.com/openstack/ceilometer/blob/stable/queens/ceilometer/pipeline/data/event_pipeline.yaml
[2] https://github.com/openstack/charm-ceilometer/blob/stable/18.11/templates/mitaka/event_pipeline.yaml#L35

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/633564

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633564
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=d398cdff364ba07a7f18f59417bb4f3af5cccbc3
Submitter: Zuul
Branch: master

commit d398cdff364ba07a7f18f59417bb4f3af5cccbc3
Author: Edward Hope-Morley <email address hidden>
Date: Mon Jan 28 17:21:18 2019 +0000

    Make event_sink publisher configurable

    The charm currently configures events to be published to
    rabbit on both the config.event_topic (event.sample queue)
    and alarm topic but as of Queens Ceilometer no longer
    consumes event.sample. This patch makes the event_sink
    publishers configurable and defaults to publishing to
    aodh to retain backwards compatibility.

    Change-Id: I5b55f31adcf2b069ff51e387a416f9f1ac4099f8
    Partial-Bug: #1676586

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/638107

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638107
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=b03fe9696d316bfa47d7f416c9671c1f50ef64f8
Submitter: Zuul
Branch: stable/18.11

commit b03fe9696d316bfa47d7f416c9671c1f50ef64f8
Author: Edward Hope-Morley <email address hidden>
Date: Mon Jan 28 17:21:18 2019 +0000

    Make event_sink publisher configurable

    The charm currently configures events to be published to
    rabbit on both the config.event_topic (event.sample queue)
    and alarm topic but as of Queens Ceilometer no longer
    consumes event.sample. This patch makes the event_sink
    publishers configurable and defaults to publishing to
    aodh to retain backwards compatibility.

    Change-Id: I5b55f31adcf2b069ff51e387a416f9f1ac4099f8
    Partial-Bug: #1676586
    (cherry picked from commit d398cdff364ba07a7f18f59417bb4f3af5cccbc3)