The memcache client that keystoneclient's WSGI middleware uses isn't thread safe, yet keystoneclient.middleware.auth_token.AuthProtocol._cache is used by every thread. In particular, what's thread-unsafe is that the memcache client has a single socket connection to the memcached server that the WSGI threads concurrently concurrently read & write to.
The consequences of this race are garbage in memcache and errors like these:
2014-03-06 14:04:56.692 16992 INFO glance.wsgi.server [-] Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 567, in __call__
token_info = self._validate_user_token(user_token)
File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 822, in _validate_user_token
self._cache_store_invalid(token_id)
File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1069, in _cache_store_invalid
self._cache_store(token_id, 'invalid')
File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 1047, in _cache_store
time=self.token_cache_time)
File "/usr/lib/python2.7/dist-packages/memcache.py", line 565, in set
return self._set("set", key, val, time, min_compress_len)
File "/usr/lib/python2.7/dist-packages/memcache.py", line 802, in _set
return _unsafe_set()
File "/usr/lib/python2.7/dist-packages/memcache.py", line 794, in _unsafe_set
server.send_cmd(fullcmd)
File "/usr/lib/python2.7/dist-packages/memcache.py", line 1112, in send_cmd
self.socket.sendall(cmd + '\r\n')
File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
tail = self.send(data, flags)
File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 302, in send
timeout_exc=socket.timeout("timed out"))
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 153, in trampoline
listener = hub.add(hub.WRITE, fileno, current.switch)
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 27, in add
listener = super(Hub, self).add(evtype, fileno, cb)
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 126, in add
evtype, fileno, evtype))
RuntimeError: Second simultaneous write on fileno 6 detected. Unless you really know what you're doing, make sure that only one greenthread can write any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)
I've attached a patch to fix the bug.
Peter, if you have a chance, can you contact me on IRC regarding this bug and the fix? (dolphm on freenode) Thanks!