Comment 17 for bug 1413341

the following test was running on each controller:
service memcached restart; service keystone restart; while [ 1 == 1 ]; do date >> memcached_node-32.txt; timeout -k 2 5 echo stats | nc 192.168.0.34 11211 >> memcached_node-32.txt; sleep 30; done
I've changed open file limits on 32 and 85 nodes
root@node-85:~# ulimit -Hn 102400
root@node-85:~# ulimit -Sn 102400

user of memcached has increased limits too
root@node-32:~# ps aux | grep memcached
root 21049 0.0 0.0 9388 940 pts/8 S+ 08:54 0:00 grep --color=auto memcached
nobody 31596 0.3 0.1 951076 52140 ? Sl Feb03 3:38 /usr/bin/memcached -P /var/run/memcached.pid -m 30535 -l 192.168.0.34 -p 11211 -U 11211 -u nobody -c 8192 -t 12
root@node-32:~# su nobody -s /bin/bash -c "ulimit -Hn"
102400
root@node-32:~# su nobody -s /bin/bash -c "ulimit -Sn"
102400
root@node-32:~# su keystone -s /bin/bash -c "ulimit -Hn"
102400
root@node-32:~# su keystone -s /bin/bash -c "ulimit -Sn"
102400

the issue has been reproduced on each controller node:
root@node-32:~# for i in 34 80 87; do echo node-$i; keystone --debug --os-token xJrbBgdH --os-endpoint http://192.168.0.$i:35357/v2.0/ user-list; done
node-34
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.34:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.34
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.34:35357/v2.0/users
node-80
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.80:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.80
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.80:35357/v2.0/users
node-87
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.87:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.87
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.87:35357/v2.0/users

here you can see that some processes didn't use a processor time
http://paste.openstack.org/show/166677/
http://paste.openstack.org/show/166678/
http://paste.openstack.org/show/166679/