Comment 5 for bug 1282865

I'm able to reproduce this consistently, and it does appear to be entirely dependent on eventlet monkey patching "thread". In my test, 100,000 requests through auth_token split between two users with different authorization, their authorization gets swapped about 0.2% of the time.

With sufficient load, python-memcached enabled in auth_token, and an unpatched thread module, eventlet produces this failure:

Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/eventlet/wsgi.py", line 414, in handle_one_response
    write('')
  File ".../lib/python2.7/site-packages/eventlet/wsgi.py", line 354, in write
    _writelines(towrite)
  File ".../python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File ".../python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File ".../lib/python2.7/site-packages/eventlet/greenio.py", line 309, in sendall
    tail = self.send(data, flags)
  File ".../lib/python2.7/site-packages/eventlet/greenio.py", line 295, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe

And python-memcached warns about unexpected responses from memcached:

  MemCached: while expecting 'STORED', got unexpected response 'VALUE tokens/acfaaff23f1576cb8485a797beb60ff4 0 1282'
  MemCached: while expecting 'STORED', got unexpected response 'END'

Given that auth_token does not depend on eventlet itself, but instead can be run by eventlet in another service, then all services depending on eventlet and relying on auth_token should be the ones to monkeypatch thread. That said, monkeypatching thread in keystoneclient/openstack/common/memorycache.py appears to resolve the problem as well.

Glance stable/havana appears to be affected:

  https://github.com/openstack/glance/blob/stable/havana/glance/cmd/registry.py#L30

Glance master does not appear to be affected:

  https://github.com/openstack/glance/blob/master/glance/cmd/registry.py#L29