os_panko missing. Ceilometer event.sample queue stuck with no consumers

Bug #1624789 reported by Paulo Matias on 2016-09-18
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
openstack-ansible
High
Unassigned

Bug Description

When setting up Ceilometer with Gnocchi as its only dispatcher, the rabbitmq /ceilometer vhost's event.sample queue is filled up with messages which are never consumed. The rabbitmq admin panel points out the queue has no consumers.

Gnocchi does not store events (it only stores metrics), however if someone doesn't deploy a db backend (nor Panko, for which we still don't have a playbook), it is dangerous that this queue might overwhelm Rabbit after getting stuck with too many messages.

If I remember well, the last week (before RC1) ceilometer-collector was consuming this queue and discarding its messages. I didn't investigate yet if the new behaviour was caused by an OSA change or by an upstream change.

Paulo Matias (paulo-matias) wrote :

This is fixed by pulling https://review.openstack.org/#/c/371244 and configuring the event pipeline as follows:

# --[cut]--

ceilometer_event_pipeline_yaml_overrides:
  sources: []
  sinks: []

# --[cut]--

https://review.openstack.org/#/c/371244 should be merged and this setting should be documented.

Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Medium
importance: Medium → High
Steve Lewis (steve-lewis) wrote :

We shouldn't use overrides in the role or integrated gate. I'd like to find what changed in Ceilometer and consider alternatives, even if it is to specify a /dev/null file dispatcher.

Paulo Matias (paulo-matias) wrote :

I'm not entirely sure the issue was not occurring before. Maybe I just didn't observe it until now.

In my point of view what is delicate about this issue is that the user might be using Gnocchi and MongoDB simultaneously, and in this case he might want to store events to MongoDB. We should probably account for this option.

All in all, I agree that asking the user to set an override is not the definite solution here. As you said, if the deployer specifies no place to store events, it makes much more sense to discard the events without requiring additional intervention from the deployer.

tags: added: newton-rc-potential
Steve Lewis (steve-lewis) wrote :

Asking the user to set an override is an option, but it seems like the current state of the world is that events should be sent to the Panko service.

Fix proposed to branch: master
Review: https://review.openstack.org/426325

Changed in openstack-ansible:
assignee: nobody → Steve Lewis (steve-lewis)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/426325
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=949df106b03e570126a882e908c80146bf768cb7
Submitter: Jenkins
Branch: master

commit 949df106b03e570126a882e908c80146bf768cb7
Author: Steve Lewis <email address hidden>
Date: Fri Jan 27 09:36:27 2017 -0800

    Update event_dispatcher when gnocchi enabled

    Per the current Ceilometer docs[1] event_dispatcher can be
    set to gnocchi.

    When gnocchi is enabled this should ensure messages are not
    accumulated endlessly in RabbitMQ.

    [1] http://docs.openstack.org/developer/ceilometer/install/dbreco.html#moving-from-ceilometer-to-gnocchi

    Change-Id: I760d135f4abd89cac3ae66f002a24a083addafb7
    Partial-Bug: 1624789

Changed in openstack-ansible:
assignee: Steve Lewis (steve-lewis) → nobody
summary: - Ceilometer event.sample queue stuck with no consumers
+ os_panko missing. Ceilometer event.sample queue stuck with no consumers
Steve Lewis (steve-lewis) wrote :

We need a role created for the panko project to resolve this isssue.

Changed in openstack-ansible:
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers