Keystone hangs trying to set a lock in Memcache

[Alexander Ignatov]

Preconditions:

1. Keystone is configured to use Memcache

backend=keystone.cache.memcache_pool
backend_argument=url:10.108.12.3,10.108.12.5,10.108.12.6
backend_argument=pool_maxsize:100

2. Memcached is deployed on each of 3 controllers.

3. 2 of 3 memcached servers are down (only the one on the primary controller is up: 10.108.12.3 is up, 10.108.12.5 and 10.108.12.6 are down)

Result: the keystone API hangs when a user do something like "keystone tenant-list". haproxy will drop the connection after 60s timeout.

strace shows that keystone tries to connect to unavailable servers in a loop, ignoring the available one.

Debug showed that keystone-all process is stuck while trying to set a lock: http://xsnippet.org/360179/ . The lock itself is implemented by setting a key in the memcache. The problem here is that python-memcache shards the keys among the configured memcached instances in a way, so that it *can* possibly fail to find an available server. This is due to how the retry logic is implemented: https://github.com/linsomniac/python-memcached/blob/master/memcache.py#L381-L396 . For a particular key (e.g. _lockusertokens-ee7e5f7374a8488bb2087e106a8834f7) 10 attempts won't be enough, and the loop within _get_server() will yield *only* unavailable memcache servers. This can be easily reproduced by http://xsnippet.org/360181/

[Roman Podoliaka (rpodolyaka)]

Looks like this happens for ~1.500 keys per 100.000 tries (~1.5%) when 1 of 3 memcache servers is available

[Roman Podoliaka (rpodolyaka)]

^ the next snippet generates 100.000 keys and prints the keys for which 10 retries won't yield an available server: http://xsnippet.org/360183/

[Irina Povolotskaya (ipovolotskaya)]

Should this be included into Release Notes?
Is there any possible workaround?

[Roman Podoliaka (rpodolyaka)]

I suggest we add something like:

Due to an issue in python-memcache library deployment of the primary controller may fail with Keystone process hanging on API requests. Should this happen, the deployment of the cluster must be restarted. Our tests show that this is a very uncommon case (like once per thousand deployments).

[Alexander Makarov (amakarov)]

Reproduced as follows:

import memcache
cl = memcache.Client(['localhost:11211', 'localhost:11212', 'localhost:11213', 'localhost:11214'])
import random
while cl.set(str(random.random()), 1): pass

'while' must be an infinite loop since cl.set() returns True on success but it exits in milliseconds instead.

[OSCI Robot (oscirobot)]

Package python-memcache has been built from changeset: http://gerrit.mirantis.com/29216
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-29216/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-29216/ubuntu /"

[OSCI Robot (oscirobot)]

Package python-memcached has been built from changeset: http://gerrit.mirantis.com/29219
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29219/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29219/centos"

[OSCI Robot (oscirobot)]

Package python-memcache has been built from changeset: http://gerrit.mirantis.com/29404
DEB Repository URL: http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-29404/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcache has been built from changeset: http://gerrit.mirantis.com/29406
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.2-stable-29406/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.2-stable-29406/ubuntu /"

[OSCI Robot (oscirobot)]

Package python-memcached has been built from changeset: http://gerrit.mirantis.com/29388
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-29388/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-29388/centos"

[OSCI Robot (oscirobot)]

Package python-memcached has been built from changeset: http://gerrit.mirantis.com/29411
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-29411/centos

[OSCI Robot (oscirobot)]

Package python-memcached-1.53-1 has been built from changeset: https://review.fuel-infra.org/460
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-460/centos

[OSCI Robot (oscirobot)]

Package python-memcached-1.53-1 has been built from changeset: https://review.fuel-infra.org/461
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-461/centos

[OSCI Robot (oscirobot)]

Package python-memcached-1.53-1 has been built from changeset: https://review.fuel-infra.org/462
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-462/centos

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/464
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.2-stable-464/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/465
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-465/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcached-1.53-1 has been built from changeset: https://review.fuel-infra.org/466
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable-466/centos

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/465
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/467
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.3-stable-467/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/468
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-468/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/483
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-483/ubuntu

[OSCI Robot (oscirobot)]

Package python-memcache-1.53-ubuntu1 has been built from changeset: https://review.fuel-infra.org/467
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.3-stable-467/ubuntu

[Alexander Makarov (amakarov)]

Patches available here:
https://review.fuel-infra.org/#/q/owner:amakarov+p:%255E.*memcache.*+status:open

[OSCI Robot (oscirobot)]

RPM package python-memcached has been built for project packages/centos6/python-memcached
Package version == 1.53, package release == 1

Changeset: https://review.fuel-infra.org/460
project: packages/centos6/python-memcached
branch: master
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: change-merged

Files placed on repository:
python-memcached-1.53-1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master/centos

[OSCI Robot (oscirobot)]

RPM package python-memcached has been built for project packages/centos6/python-memcached
Package version == 1.53, package release == 1

Changeset: https://review.fuel-infra.org/462
project: packages/centos6/python-memcached
branch: 5.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: change-merged

Files placed on repository:
python-memcached-1.53-1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable/centos

[OSCI Robot (oscirobot)]

RPM package python-memcached has been built for project packages/centos6/python-memcached
Package version == 1.53, package release == 1

Changeset: https://review.fuel-infra.org/466
project: packages/centos6/python-memcached
branch: 5.0.3
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: change-merged

Files placed on repository:
python-memcached-1.53-1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable/centos

[OSCI Robot (oscirobot)]

DEB package python-memcache has been built for project packages/precise/python-memcache
Package version == 1.53, package release == ubuntu1

Changeset: https://review.fuel-infra.org/467
project: packages/precise/python-memcache
branch: 5.0.3
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: change-merged

Files placed on repository:
python-memcache_1.53-ubuntu1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.3-stable/ubuntu

[OSCI Robot (oscirobot)]

DEB package python-memcache has been built for project packages/precise/python-memcache
Package version == 1.53, package release == ubuntu1

Changeset: https://review.fuel-infra.org/468
project: packages/precise/python-memcache
branch: master
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: change-merged

Files placed on repository:
python-memcache_1.53-ubuntu1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-master/ubuntu

[OSCI Robot (oscirobot)]

RPM package python-memcached has been built for project packages/centos6/python-memcached
Package version == 1.53, package release == 1

Changeset: https://review.fuel-infra.org/832
project: packages/centos6/python-memcached
branch: 6.0
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: patchset-created

Files placed on repository:
python-memcached-1.53-1.mira1.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-832/centos

[OSCI Robot (oscirobot)]

RPM package python-memcached has been built for project packages/centos6/python-memcached
Package version == 1.53, package release == 1

Changeset: https://review.fuel-infra.org/832
project: packages/centos6/python-memcached
branch: 6.0
author: Alexander Makarov
committer: Alexander Makarov
subject: Applying get_server_fix.patch
status: change-merged

Files placed on repository:
python-memcached-1.53-1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos