os_panko missing. Ceilometer event.sample queue stuck with no consumers

[Paulo Matias]

When setting up Ceilometer with Gnocchi as its only dispatcher, the rabbitmq /ceilometer vhost's event.sample queue is filled up with messages which are never consumed. The rabbitmq admin panel points out the queue has no consumers.

Gnocchi does not store events (it only stores metrics), however if someone doesn't deploy a db backend (nor Panko, for which we still don't have a playbook), it is dangerous that this queue might overwhelm Rabbit after getting stuck with too many messages.

If I remember well, the last week (before RC1) ceilometer-collector was consuming this queue and discarding its messages. I didn't investigate yet if the new behaviour was caused by an OSA change or by an upstream change.

[Paulo Matias (paulo-matias)]

This is fixed by pulling https://review.openstack.org/#/c/371244 and configuring the event pipeline as follows:

# --[cut]--

ceilometer_event_pipeline_yaml_overrides:
  sources: []
  sinks: []

# --[cut]--

https://review.openstack.org/#/c/371244 should be merged and this setting should be documented.

[Steve Lewis (steve-lewis)]

We shouldn't use overrides in the role or integrated gate. I'd like to find what changed in Ceilometer and consider alternatives, even if it is to specify a /dev/null file dispatcher.

[Paulo Matias (paulo-matias)]

I'm not entirely sure the issue was not occurring before. Maybe I just didn't observe it until now.

In my point of view what is delicate about this issue is that the user might be using Gnocchi and MongoDB simultaneously, and in this case he might want to store events to MongoDB. We should probably account for this option.

All in all, I agree that asking the user to set an override is not the definite solution here. As you said, if the deployer specifies no place to store events, it makes much more sense to discard the events without requiring additional intervention from the deployer.

[Steve Lewis (steve-lewis)]

Asking the user to set an override is an option, but it seems like the current state of the world is that events should be sent to the Panko service.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/426325

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/426325
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=949df106b03e570126a882e908c80146bf768cb7
Submitter: Jenkins
Branch: master

commit 949df106b03e570126a882e908c80146bf768cb7
Author: Steve Lewis <email address hidden>
Date: Fri Jan 27 09:36:27 2017 -0800

    Update event_dispatcher when gnocchi enabled

    Per the current Ceilometer docs[1] event_dispatcher can be
    set to gnocchi.

    When gnocchi is enabled this should ensure messages are not
    accumulated endlessly in RabbitMQ.

    [1] http://docs.openstack.org/developer/ceilometer/install/dbreco.html#moving-from-ceilometer-to-gnocchi

    Change-Id: I760d135f4abd89cac3ae66f002a24a083addafb7
    Partial-Bug: 1624789

[Steve Lewis (steve-lewis)]

We need a role created for the panko project to resolve this isssue.

[Jesse Pretorius (jesse-pretorius)]

There is now a Panko role available: https://github.com/openstack/openstack-ansible-os_panko

[Dmitriy Rabotyagov (noonedeadpunk)]

I guess since we have panko role, this bug can be considered as resolved.