alarm evaluator fails because unicode object not callable

[Mike Spreitzer]

In a recent install by DevStack, I find that the alarm evaluator suffers an error on every attempt to evaluate an alarm. Here is the log output for one such failure:

2014-09-03 04:34:01.686 12404 DEBUG ceilometerclient.openstack.common.apiclient.client [-] Request returned failure status: 401 request /opt/stack/python-ceilometerclient/ceilometerclient/op\
enstack/common/apiclient/client.py:181
2014-09-03 04:34:01.687 12404 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service Traceback (most recent call last):
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/ceilometer/ceilometer/alarm/service.py", line 89, in _evaluate_assigned_alarms
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/ceilometer/ceilometer/alarm/service.py", line 132, in _assigned_alarms
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service 'value': True}])
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/v2/alarms.py", line 80, in list
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service return self._list(options.build_url(self._path(), q))
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/common/base.py", line 58, in _list
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service body = self.api.get(url).json()
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/openstack/common/apiclient/client.py", line 328, in get
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service return self.client_request("GET", url, **kwargs)
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/openstack/common/apiclient/client.py", line 322, in client_request
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service self, method, url, **kwargs)
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/openstack/common/apiclient/client.py", line 248, in client_request
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service self.authenticate()
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/openstack/common/apiclient/client.py", line 292, in authenticate
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service self.auth_plugin.authenticate(self)
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/openstack/common/apiclient/auth.py", line 194, in authenticate
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service self._do_authenticate(http_client)
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service File "/opt/stack/python-ceilometerclient/ceilometerclient/client.py", line 186, in _do_authenticate
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service self.opts['token'] = token()
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service TypeError: 'unicode' object is not callable
2014-09-03 04:34:01.687 12404 TRACE ceilometer.alarm.service
2014-09-03 04:35:01.897 12404 DEBUG ceilometerclient.openstack.common.apiclient.client [-] Request returned failure status: 401 request /opt/stack/python-ceilometerclient/ceilometerclient/openstack/common/apiclient/client.py:181

Here are the versions I am running:

ubuntu@mjs-dstk-902c:/opt/stack/ceilometer$ git branch -v
* master b7e71b5 Merge "Imported Translations from Transifex"

ubuntu@mjs-dstk-902c:/opt/stack/ceilometer$ cd ../python-ceilometerclient/

ubuntu@mjs-dstk-902c:/opt/stack/python-ceilometerclient$ git branch -v
* master 4cef2bf Merge "Revamp documentation, add module references"

ubuntu@mjs-dstk-902c:/opt/stack/python-ceilometerclient$ cd ../python-keystoneclient/

ubuntu@mjs-dstk-902c:/opt/stack/python-keystoneclient$ git remote -v
origin https://review.openstack.org/openstack/python-keystoneclient (fetch)
origin https://review.openstack.org/openstack/python-keystoneclient (push)

ubuntu@mjs-dstk-902c:/opt/stack/python-keystoneclient$ git branch -v
* (detached from FETCH_HEAD) 76e3e6b Using correct keyword for region in v3
  master 1643f7d Fix test mistake with requests-mock

I installed patch set 2 of https://review.openstack.org/#/c/118383/ for keystone client, installed everything else from HEAD.

[Sergey Kraynev (skraynev)]

Mike, thanks that raise this problem. I saw the same thing in my devstack. And I solved it by restarting ceilometer-alarm-evaluator service, but it come back after some time and I have not any idea what is the reason.

Also I got it before creating fix for https://bugs.launchpad.net/keystone/+bug/1364463. So I think it's unrelated problem.

[Mike Spreitzer (mike-spreitzer)]

This is very strange! It looks like the fix (https://review.openstack.org/#/c/58051/) to https://bugs.launchpad.net/python-ceilometerclient/+bug/1254186 has somehow been undone.

[Mike Spreitzer (mike-spreitzer)]

It looks to me like things went wrong at https://review.openstack.org/#/c/68939/

That change was created in January and merged in August. In between those two events, https://bugs.launchpad.net/python-ceilometerclient/+bug/1254186 was fixed in March. The merge in August undid the fix from March.

[Dina Belova (dbelova)]

Folks, this error is the known issue, we're fixing it now in the oslo.incubator and later will provide the fix to the ceilometerclient itself