It appears that on a keystone authentication failure, cinder client (accessed by Congress) threw an unanticipated error. See following Congress log and corresponding part of cinder api log.
Low priority because it only happened on the very infrequent auth failure (1 occurrence observed in three days).
python-cinderclient version: 1.9.0
Devstack version: commit 81d89cf3584a5edadbaa2514305cf5721b29cdff
Congress version: commit b510c1171c777433cc4dd6260316371309bfd5d2; Merge: 3c69a14 83ee990
Cinder version: commit 4e2e29d1e55645bb83120e156e0f2819e925f745; Merge: a956bb5 642a1c7
Congress log:
2016-09-25 06:39:11.210 ERROR congress.datasources.datasource_driver [-] Datasource driver raised exception
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver Traceback (most recent call last):
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/datasource_driver.py", line 1384, in poll
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver self.update_from_datasource() # sets self.state
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/datasource_driver.py", line 1371, in update_from_datasource
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver self.update_methods[registered_table]()
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/opt/stack/congress/congress/datasources/cinder_driver.py", line 113, in <lambda>
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver self.cinder_client.services.list(host=None, binary=None))
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/v3/services.py", line 47, in list
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver return self._list(url, "services")
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/base.py", line 83, in _list
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver resp, body = self.api.client.get(url)
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 143, in get
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver return self._cs_request(url, 'GET', **kwargs)
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 134, in _cs_request
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver return self.request(url, method, **kwargs)
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 123, in request
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver raise exceptions.from_response(resp, body)
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver File "/usr/local/lib/python2.7/dist-packages/cinderclient/exceptions.py", line 246, in from_response
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver message = error.get('message', message)
2016-09-25 06:39:11.210 TRACE congress.datasources.datasource_driver AttributeError: 'unicode' object has no attribute 'get'
Cinder log:
2016-09-25 06:39:11.180 ERROR keystonemiddleware.auth_token [req-a411a1a7-3788-4554-8203-c43c375e8917 f3be9a82aea8480c9da986daf672d002 a9497e71e236456d87ebc284c348400d] Bad response code while validating token: 503
2016-09-25 06:39:11.185 WARNING keystonemiddleware.auth_token [req-a411a1a7-3788-4554-8203-c43c375e8917 f3be9a82aea8480c9da986daf672d002 a9497e71e236456d87ebc284c348400d] Identity response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2016-09-25 06:39:11.188 CRITICAL keystonemiddleware.auth_token [req-a411a1a7-3788-4554-8203-c43c375e8917 f3be9a82aea8480c9da986daf672d002 a9497e71e236456d87ebc284c348400d] Unable to validate token: Failed to fetch token data from identity server
2016-09-25 06:39:11.210 INFO eventlet.wsgi.server [req-a411a1a7-3788-4554-8203-c43c375e8917 f3be9a82aea8480c9da986daf672d002 a9497e71e236456d87ebc284c348400d] 192.168.218.145 "GET /v2/a9497e71e236456d87ebc284c348400d/os-services HTTP/1.1" status: 503 len: 401 time: 0.5045660
