Mirantis OpenStack

Comment 2 for bug 1413341

Revision history for this message

Alexander Makarov (amakarov) wrote on 2015-01-22: Re: keystone doesn't response if cluster is actively used

Nova logs:

2015-01-21T05:48:05.160002+00:00 debug: REQ: curl -i -X GET http://192.168.0.2:35357/v3/auth/tokens -H "X-Subject-Token: TOKEN_REDACTED" -H "User-Agent: python-keystoneclient" -H "Accept: a
pplication/json" -H "X-Auth-Token: TOKEN_REDACTED"
2015-01-21T05:48:05.160002+00:00 debug: RESP:
2015-01-21T05:48:05.160002+00:00 debug: Request returned failure status: 504
2015-01-21T05:48:05.160002+00:00 err: Bad response code while validating token: 504
2015-01-21T05:48:05.160002+00:00 warning: Identity response: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
2015-01-21T05:48:05.160002+00:00 warning: Authorization failed for token
2015-01-21T05:48:05.160002+00:00 info: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 459, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 390, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 357, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 340, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

Meanwhile keystone log reports nothing.
My suspicion is: keystone locks for too long under high load as memcache lock in it sleeps ~0.5sec for every of 15 acquire attempts.

We can try to reduce this value (to 0 for ex.) and wait for "Maximum lock attempts on %s occurred." in keystone log.