I'm able to reproduce this consistently, and it does appear to be entirely dependent on eventlet monkey patching "thread". In my test, 100,000 requests through auth_token split between two users with different authorization, their authorization gets swapped about 0.2% of the time.
With sufficient load, python-memcached enabled in auth_token, and an unpatched thread module, eventlet produces this failure:
Traceback (most recent call last):
File ".../lib/python2.7/site-packages/eventlet/wsgi.py", line 414, in handle_one_response
write('')
File ".../lib/python2.7/site-packages/eventlet/wsgi.py", line 354, in write
_writelines(towrite)
File ".../python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 334, in writelines
self.flush()
File ".../python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
File ".../lib/python2.7/site-packages/eventlet/greenio.py", line 309, in sendall
tail = self.send(data, flags)
File ".../lib/python2.7/site-packages/eventlet/greenio.py", line 295, in send
total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe
And python-memcached warns about unexpected responses from memcached:
MemCached: while expecting 'STORED', got unexpected response 'VALUE tokens/acfaaff23f1576cb8485a797beb60ff4 0 1282'
MemCached: while expecting 'STORED', got unexpected response 'END'
Given that auth_token does not depend on eventlet itself, but instead can be run by eventlet in another service, then all services depending on eventlet and relying on auth_token should be the ones to monkeypatch thread. That said, monkeypatching thread in keystoneclient/openstack/common/memorycache.py appears to resolve the problem as well.
I'm able to reproduce this consistently, and it does appear to be entirely dependent on eventlet monkey patching "thread". In my test, 100,000 requests through auth_token split between two users with different authorization, their authorization gets swapped about 0.2% of the time.
With sufficient load, python-memcached enabled in auth_token, and an unpatched thread module, eventlet produces this failure:
Traceback (most recent call last): python2. 7/site- packages/ eventlet/ wsgi.py" , line 414, in handle_one_response python2. 7/site- packages/ eventlet/ wsgi.py" , line 354, in write (towrite) 2.7.6/Framework s/Python. framework/ Versions/ 2.7/lib/ python2. 7/socket. py", line 334, in writelines 2.7.6/Framework s/Python. framework/ Versions/ 2.7/lib/ python2. 7/socket. py", line 303, in flush _sock.sendall( view[write_ offset: write_offset+ buffer_ size]) python2. 7/site- packages/ eventlet/ greenio. py", line 309, in sendall python2. 7/site- packages/ eventlet/ greenio. py", line 295, in send data[total_ sent:], flags)
File ".../lib/
write('')
File ".../lib/
_writelines
File ".../python/
self.flush()
File ".../python/
self.
File ".../lib/
tail = self.send(data, flags)
File ".../lib/
total_sent += fd.send(
error: [Errno 32] Broken pipe
And python-memcached warns about unexpected responses from memcached:
MemCached: while expecting 'STORED', got unexpected response 'VALUE tokens/ acfaaff23f1576c b8485a797beb60f f4 0 1282'
MemCached: while expecting 'STORED', got unexpected response 'END'
Given that auth_token does not depend on eventlet itself, but instead can be run by eventlet in another service, then all services depending on eventlet and relying on auth_token should be the ones to monkeypatch thread. That said, monkeypatching thread in keystoneclient/ openstack/ common/ memorycache. py appears to resolve the problem as well.
Glance stable/havana appears to be affected:
https:/ /github. com/openstack/ glance/ blob/stable/ havana/ glance/ cmd/registry. py#L30
Glance master does not appear to be affected:
https:/ /github. com/openstack/ glance/ blob/master/ glance/ cmd/registry. py#L29