Ceilometer agent polling isn't working in TripleO with TLS enabled

Bug #1764451 reported by Juan Antonio Osorio Robles on 2018-04-16
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceilometer
Critical
Unassigned
tripleo
High
Unassigned

Bug Description

With TLS enabled in TripleO, the ceilometer agent is getting the following errors:

2018-04-16 15:44:33.399 25 INFO panko.storage.impl_sqlalchemy [-] Duplicate event detected, skipping it: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '37aa2f20-bd04-40c7-b321-65a38df61669' for key 'mess
age_id'") [SQL: u'INSERT INTO event (message_id, `generated`, raw, event_type_id) VALUES (%(message_id)s, %(generated)s, %(raw)s, %(event_type_id)s)'] [parameters: {'raw': '{}', 'generated': Decimal('1523893473.
370949'), 'event_type_id': 81, 'message_id': u'37aa2f20-bd04-40c7-b321-65a38df61669'}] (Background on this error at: http://sqlalche.me/e/gkpj)
2018-04-16 15:44:33.829 25 INFO panko.storage.impl_sqlalchemy [-] Duplicate event detected, skipping it: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'f2805c3f-2384-4c2b-b006-9c41c3d49f36' for key 'mess
age_id'") [SQL: u'INSERT INTO event (message_id, `generated`, raw, event_type_id) VALUES (%(message_id)s, %(generated)s, %(raw)s, %(event_type_id)s)'] [parameters: {'raw': '{}', 'generated': Decimal('1523893473.
776865'), 'event_type_id': 82, 'message_id': u'f2805c3f-2384-4c2b-b006-9c41c3d49f36'}] (Background on this error at: http://sqlalche.me/e/gkpj)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event [-] Fail to process a notification: KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event Traceback (most recent call last):
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/pipeline/event.py", line 59, in process_notifications
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event event = self.event_converter.to_event(priority, message)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 276, in to_event
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event return edef.to_event(priority, notification_body)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 152, in to_event
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event traits = [trait for trait in traits if trait is not None]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 150, in <genexpr>
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event for t in self.traits)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 58, in to_trait
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event value = self.parse(notification_body)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/ceilometer/declarative.py", line 116, in parse
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event values = self.getter(obj)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 226, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event for subdata in self.left.find(datum)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 228, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event for submatch in self.right.find(subdata)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 444, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.event
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample [-] Fail to process notification: KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample Traceback (most recent call last):
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/pipeline/sample.py", line 46, in process_notifications
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample p(list(self.build_sample(message)))
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/meter/notifications.py", line 230, in build_sample
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample for s in d.to_samples(notification):
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/meter/notifications.py", line 124, in to_samples
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample value = parser.parse(message)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/ceilometer/declarative.py", line 116, in parse
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample values = self.getter(obj)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 226, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample for subdata in self.left.find(datum)
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 228, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample for submatch in self.right.find(subdata)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 444, in find
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample KeyError: 0
2018-04-16 15:44:34.179 25 ERROR ceilometer.pipeline.sample

Changed in ceilometer:
importance: Undecided → Critical

Note that this is happening in a very basic setup without network isolation.

Julien Danjou (jdanjou) wrote :

The fact that you receive the event twice as indicated Panko is quite suspicious on your setup.

This bug in TripleO seems to present a similar failure in tempest but without the TLS setup https://bugs.launchpad.net/tripleo/+bug/1764660

Changed in tripleo:
status: New → Confirmed
importance: Undecided → High
milestone: none → rocky-1
Changed in tripleo:
status: Confirmed → Triaged
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers