In the periodic task to sync cluster statuses, we catch [1] an exception when the stack doesn't exist.
However the heat client throws an exception because the trust is already deleted.
We need to catch the authentication exception too.
2017-10-26 15:12:56.275 20618 ERROR magnum.common.keystone [req-9997650a-47da-41cc-8211-3afbd9071403 - - - 4cb76a98145b11e793ae92361f002671 -] Keystone API connection failed: no password, trust_id or token found. 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall [req-9997650a-47da-41cc-8211-3afbd9071403 - - - 4cb76a98145b11e793ae92361f002671 -] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: AuthorizationFailure: reason Keystone API connection failed: no password, trust_id or token found. 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall Traceback (most recent call last): 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/service/periodic.py", line 70, in update_status 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall cdriver.update_cluster_status(self.ctx, self.cluster) 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/drivers/heat/driver.py", line 83, in update_cluster_status 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall poller.poll_and_check() 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/drivers/heat/driver.py", line 172, in poll_and_check 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall stack = self.openstack_client.heat().stacks.get( 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/common/exception.py", line 57, in wrapped 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall return func(*args, **kw) 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/common/clients.py", line 93, in heat 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall region_name=region_name) 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/common/clients.py", line 44, in url_for 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall return self.keystone().session.get_endpoint(**kwargs) 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/common/keystone.py", line 57, in session 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall auth = self._get_auth() 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/magnum/common/keystone.py", line 97, in _get_auth 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall message='reason %s' % msg) 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall AuthorizationFailure: reason Keystone API connection failed: no password, trust_id or token found. 2017-10-26 15:12:56.275 20618 ERROR oslo.service.loopingcall
[1] http://git.openstack.org/cgit/openstack/magnum/tree/magnum/drivers/heat/driver.py#n187
In the periodic task to sync cluster statuses, we catch [1] an exception when the stack doesn't exist.
However the heat client throws an exception because the trust is already deleted.
We need to catch the authentication exception too.
2017-10-26 15:12:56.275 20618 ERROR magnum. common. keystone [req-9997650a- 47da-41cc- 8211-3afbd90714 03 - - - 4cb76a98145b11e 793ae92361f0026 71 -] Keystone API connection failed: no password, trust_id or token found. loopingcall [req-9997650a- 47da-41cc- 8211-3afbd90714 03 - - - 4cb76a98145b11e 793ae92361f0026 71 -] Fixed interval looping call 'magnum. service. periodic. ClusterUpdateJo b.update_ status' failed: AuthorizationFa ilure: reason Keystone API connection failed: no password, trust_id or token found. loopingcall Traceback (most recent call last): loopingcall File "/usr/lib/ python2. 7/site- packages/ oslo_service/ loopingcall. py", line 137, in _run_loop loopingcall result = func(*self.args, **self.kw) loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ service/ periodic. py", line 70, in update_status loopingcall cdriver. update_ cluster_ status( self.ctx, self.cluster) loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ drivers/ heat/driver. py", line 83, in update_ cluster_ status loopingcall poller. poll_and_ check() loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ drivers/ heat/driver. py", line 172, in poll_and_check loopingcall stack = self.openstack_ client. heat(). stacks. get( loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ common/ exception. py", line 57, in wrapped loopingcall return func(*args, **kw) loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ common/ clients. py", line 93, in heat loopingcall region_ name=region_ name) loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ common/ clients. py", line 44, in url_for loopingcall return self.keystone( ).session. get_endpoint( **kwargs) loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ common/ keystone. py", line 57, in session loopingcall auth = self._get_auth() loopingcall File "/usr/lib/ python2. 7/site- packages/ magnum/ common/ keystone. py", line 97, in _get_auth loopingcall message='reason %s' % msg) loopingcall AuthorizationFa ilure: reason Keystone API connection failed: no password, trust_id or token found. loopingcall
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
2017-10-26 15:12:56.275 20618 ERROR oslo.service.
[1] http:// git.openstack. org/cgit/ openstack/ magnum/ tree/magnum/ drivers/ heat/driver. py#n187