From this commit [1] nova now needs access to keystone to perform quota (this bug is mostly related to issue we had with quota update).
When keystone is not available the nova-api (running in eventlet) tries to use the endpoints ordered in [keystone]/valid_interfaces, we did not have access to the internal endpoint which caused this issue:
2018-05-14 15:54:46.134 1241 INFO nova.api.openstack.identity [req-8b383cf0-7f99-41e6-9de3-5e694fb24449 f13940ac09924d8582fe6612e838c7a7 9387d3a7be2a487784a90660b6e182cb - default default] Unable to contact keystone to verify project_id
You'll also see:
2018-05-14 15:54:46.419 1241 INFO nova.osapi_compute.wsgi.server [req-e9da4d33-05be-42fe-891d-0d201d2e8311 83e8a17bf7874682a86f9aa58f4c9507 e83ea76e472f48679f6fa6070a8a16e1 - default default] Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 512, in handle_one_response
write(b''.join(towrite))
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 453, in write
wfile.flush()
File "/usr/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 385, in sendall
tail = self.send(data, flags)
File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 379, in send
return self._send_loop(self.fd.send, data, flags)
File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 366, in _send_loop
return send_method(data, *args)
error: [Errno 104] Connection reset by peer
Now this is correct, however what happens next is imo not correct, it generates a 200 OK response when it actually failed to perform the requested action.
From this commit [1] nova now needs access to keystone to perform quota (this bug is mostly related to issue we had with quota update).
When keystone is not available the nova-api (running in eventlet) tries to use the endpoints ordered in [keystone] /valid_ interfaces, we did not have access to the internal endpoint which caused this issue:
2018-05-14 15:54:46.134 1241 INFO nova.api. openstack. identity [req-8b383cf0- 7f99-41e6- 9de3-5e694fb244 49 f13940ac09924d8 582fe6612e838c7 a7 9387d3a7be2a487 784a90660b6e182 cb - default default] Unable to contact keystone to verify project_id
You'll also see:
2018-05-14 15:54:46.419 1241 INFO nova.osapi_ compute. wsgi.server [req-e9da4d33- 05be-42fe- 891d-0d201d2e83 11 83e8a17bf787468 2a86f9aa58f4c95 07 e83ea76e472f486 79f6fa6070a8a16 e1 - default default] Traceback (most recent call last): python2. 7/dist- packages/ eventlet/ wsgi.py" , line 512, in handle_one_response b''.join( towrite) ) python2. 7/dist- packages/ eventlet/ wsgi.py" , line 453, in write python2. 7/socket. py", line 307, in flush _sock.sendall( view[write_ offset: write_offset+ buffer_ size]) python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 385, in sendall python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 379, in send loop(self. fd.send, data, flags) python2. 7/dist- packages/ eventlet/ greenio/ base.py" , line 366, in _send_loop
File "/usr/lib/
write(
File "/usr/lib/
wfile.flush()
File "/usr/lib/
self.
File "/usr/lib/
tail = self.send(data, flags)
File "/usr/lib/
return self._send_
File "/usr/lib/
return send_method(data, *args)
error: [Errno 104] Connection reset by peer
Now this is correct, however what happens next is imo not correct, it generates a 200 OK response when it actually failed to perform the requested action.
2018-05-14 15:54:46.420 1241 INFO nova.osapi_ compute. wsgi.server [req-e9da4d33- 05be-42fe- 891d-0d201d2e83 11 83e8a17bf787468 2a86f9aa58f4c95 07 e83ea76e472f486 79f6fa6070a8a16 e1 - default default] ::ffff: 195.74. 38.54,172. 20.104. 11 "PUT /v2/e83ea76e472 f48679f6fa6070a 8a16e1/ os-quota- sets/e83ea76e47 2f48679f6fa6070 a8a16e1 HTTP/1.1" status: 200 len: 0 time: 128.0125880
For us we were able to notice this with 504 gateway error because the time the request took (128 seconds) was too long for our load balancer to allow.
I think atleast catching the exception and setting the return code to 500 would be appropriate, and also output it as an error and not a INFO message.
[1] https:/ /github. com/openstack/ nova/commit/ 1f120b5649ba03a a5b2490a82c08b7 7c580f12d7