OK I turned on keystonemiddleware DEBUG and get the below stack trace, I also ran it through PDB and it fails when it's getting the revocation list. It works the first time and then requests work, when it goes to get it again it the consistently fails from then onwards.

Keystone is definitely working and responding to requests. The weird thing is that restarting nova-api fixes it and will work for several requests before going back to the broken state.

 


2015-03-31 10:01:41.042 30520 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-31 10:01:41.545 30520 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-31 10:01:42.547 30520 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-31 10:01:44.549 30520 ERROR keystonemiddleware.auth_token [-] HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-31 10:01:44.550 30520 DEBUG keystonemiddleware.auth_token [-] Token validation failure. _validate_user_token /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:922
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token Traceback (most recent call last):
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 913, in _validate_user_token
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     verified = self._verify_signed_token(user_token, token_ids)
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 1185, in _verify_signed_token
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     if self._is_signed_token_revoked(token_ids):
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 1137, in _is_signed_token_revoked
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     if self._is_token_id_in_revoked_list(token_id):
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 1144, in _is_token_id_in_revoked_list
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     revocation_list = self._token_revocation_list
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 1251, in _token_revocation_list
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     self._token_revocation_list = self._fetch_revocation_list()
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 1286, in _fetch_revocation_list
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     additional_headers=headers)
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 828, in _json_request
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     response = self._http_request(method, path, **kwargs)
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 795, in _http_request
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token     raise NetworkError('Unable to communicate with keystone')
2015-03-31 10:01:44.550 30520 TRACE keystonemiddleware.auth_token NetworkError: Unable to communicate with keystone