Ceilometer alarm issue resurfaced probably because of changes to ceilometer. An error is expected when asking ceilometer for alarms while aodh is not present, which our driver code attempts to suppress, but ceilometer now generates a new kind of error. Given that ceilometer's handling of this case has proven to be unstable, now may be the time to remove the alarms table from ceilometer completely.
2017-01-18 10:17:50.933 ERROR congress.datasources.datasource_driver [-] Datasource driver raised exception
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver Traceback (most recent call last):
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/datasource_driver.py", line 1388, in poll
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver self.update_from_datasource() # sets self.state
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/datasource_driver.py", line 1375, in update_from_datasource
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver self.update_methods[registered_table]()
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/ceilometer_driver.py", line 194, in <lambda>
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver alarms_list_suppress_no_aodh_error(self.ceilometer_client))
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/ceilometer_driver.py", line 182, in alarms_list_suppress_no_aodh_error
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver return self.ceilometer_client.alarms.list()
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/alarms.py", line 83, in list
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver return self._list(options.build_url(self._path(), q))
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/common/base.py", line 63, in _list
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver resp = self.api.get(url)
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 216, in get
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver return self.request(url, 'GET', **kwargs)
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 473, in request
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver raise exc.from_response(resp, body)
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
2017-01-18 10:17:50.933 TRACE congress.datasources.datasource_driver
One more thought on this bug:
If we're sticking to the deprecation policy strictly, it's too early to remove the table completely in Ocata. The big problem is not that errors are being generated in the logs when aodh isn't running; that's actually a good thing to let the deployer know. The real problem is that when updating alarms table fails, the update process is aborted and subsequent tables are not updated.
One idea is to change the try-except structure in the polling code so that error on one table update does not stop subsequent tables from being updated. Seems that's a good thing anyway regardless of this particular issue.