Ceilometer

Ceilometer agent polling isn't working in TripleO with TLS enabled

Bug #1764451 reported by Juan Antonio Osorio Robles
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceilometer
Invalid
Critical
Unassigned
tripleo
Invalid
High
Unassigned
tripleo stein-3

Bug Description

With TLS enabled in TripleO, the ceilometer agent is getting the following errors:

2018-04-16 15:44:33.399 25 INFO panko.storage.impl_sqlalchemy [-] Duplicate event detected, skipping it: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '37aa2f20-bd04-40c7-b321-65a38df61669' for key 'mess
age_id'") [SQL: u'INSERT INTO event (message_id, `generated`, raw, event_type_id) VALUES (%(message_id)s, %(generated)s, %(raw)s, %(event_type_id)s)'] [parameters: {'raw': '{}', 'generated': Decimal('1523893473.
370949'), 'event_type_id': 81, 'message_id': u'37aa2f20-bd04-40c7-b321-65a38df61669'}] (Background on this error at: http://sqlalche.me/e/gkpj)
2018-04-16 15:44:33.829 25 INFO panko.storage.impl_sqlalchemy [-] Duplicate event detected, skipping it: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'f2805c3f-2384-4c2b-b006-9c41c3d49f36' for key 'mess
age_id'") [SQL: u'INSERT INTO event (message_id, `generated`, raw, event_type_id) VALUES (%(message_id)s, %(generated)s, %(raw)s, %(event_type_id)s)'] [parameters: {'raw': '{}', 'generated': Decimal('1523893473.
776865'), 'event_type_id': 82, 'message_id': u'f2805c3f-2384-4c2b-b006-9c41c3d49f36'}] (Background on this error at: http://sqlalche.me/e/gkpj)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event [-] Fail to process a notification: KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event Traceback (most recent call last):
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/pipeline/event.py", line 59, in process_notifications
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event event = self.event_converter.to_event(priority, message)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 276, in to_event
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event return edef.to_event(priority, notification_body)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 152, in to_event
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event traits = [trait for trait in traits if trait is not None]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 150, in <genexpr>
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event for t in self.traits)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 58, in to_trait
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event value = self.parse(notification_body)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/declarative.py", line 116, in parse
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event values = self.getter(obj)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 226, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event for subdata in self.left.find(datum)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 228, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event for submatch in self.right.find(subdata)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 444, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample [-] Fail to process notification: KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample Traceback (most recent call last):
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/pipeline/sample.py", line 46, in process_notifications
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample p(list(self.build_sample(message)))
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/meter/notifications.py", line 230, in build_sample
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample for s in d.to_samples(notification):
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/meter/notifications.py", line 124, in to_samples
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample value = parser.parse(message)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/declarative.py", line 116, in parse
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample values = self.getter(obj)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 226, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample for subdata in self.left.find(datum)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 228, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample for submatch in self.right.find(subdata)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 444, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample

Pradeep Kilambi (pkilambi)
Changed in ceilometer:
importance: Undecided → Critical
Revision history for this message
Juan Antonio Osorio Robles (juan-osorio-robles) wrote :

Note that this is happening in a very basic setup without network isolation.

Revision history for this message
Julien Danjou (jdanjou) wrote :

The fact that you receive the event twice as indicated Panko is quite suspicious on your setup.

Revision history for this message
Juan Antonio Osorio Robles (juan-osorio-robles) wrote :

This bug in TripleO seems to present a similar failure in tempest but without the TLS setup https://bugs.launchpad.net/tripleo/+bug/1764660

Revision history for this message
Juan Antonio Osorio Robles (juan-osorio-robles) wrote :

For reference, here are the tempest logs of the ceilometer bug I originally reported http://logs.openstack.org/52/560752/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/b886ebf/logs/undercloud/home/zuul/tempest/tempest.html.gz

Juan Antonio Osorio Robles (juan-osorio-robles)
Changed in tripleo:
status: New → Confirmed
importance: Undecided → High
milestone: none → rocky-1
Emilien Macchi (emilienm)
Changed in tripleo:
status: Confirmed → Triaged
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: rocky-1 → rocky-2
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: rocky-2 → rocky-3
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Juan Antonio Osorio Robles (juan-osorio-robles)
Changed in tripleo:
milestone: stein-1 → stein-2
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: stein-2 → stein-3
Revision history for this message
Juan Antonio Osorio Robles (juan-osorio-robles) wrote :

This is no longer an issue.

Changed in tripleo:
status: Triaged → Invalid
Juan Antonio Osorio Robles (juan-osorio-robles)
Changed in ceilometer:
status: New → Invalid
Revision history for this message
cristi1979 (cristi-falcas) wrote :
Download full text (5.5 KiB)

How I can investigate this error?

I'm getting it too:

```
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event [-] Fail to process a notification: KeyError: 0
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event Traceback (most recent call last):
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/pipeline/event.py", line 59, in process_notifications
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event event = self.event_converter.to_event(priority, message)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/event/converter.py", line 276, in to_event
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event return edef.to_event(priority, notification_body)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/event/converter.py", line 152, in to_event
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event traits = [trait for trait in traits if trait is not None]
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/event/converter.py", line 152, in <listcomp>
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event traits = [trait for trait in traits if trait is not None]
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/event/converter.py", line 150, in <genexpr>
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event for t in self.traits)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/event/converter.py", line 58, in to_trait
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event value = self.parse(notification_body)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/ceilometer/declarative.py", line 116, in parse
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event values = self.getter(obj)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/jsonpath_rw/jsonpath.py", line 226, in find
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event for subdata in self.left.find(datum)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/jsonpath_rw/jsonpath.py", line 226, in find
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event for subdata in self.left.find(datum)
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/jsonpath_rw/jsonpath.py", line 228, in <listcomp>
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event for submatch in self.right.find(subdata)]
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event File "/usr/lib/python3/dist-packages/jsonpath_rw/jsonpath.py", line 444, in find
2019-04-08 18:04:50.010 15608 ERROR ceilometer.pipeline.event return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2019-04-08 18:04:50.010 15608 ...

Read more...

Revision history for this message
cristi1979 (cristi-falcas) wrote :

Please ignore me. It was a file publisher with a wrong path

Revision history for this message
Chiawei Xie (dommgifer) wrote :

@cristi1979

I have same issue.

which file with a wrong path?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.