keystone behavior when one memcache backend is down

[Sergii Golovatiuk]

Hi,

Our implementation uses dogpile.cache.memcached as a backend for tokens. Recently, I have found interesting behavior when one of memcache regions went down. There is a 3-6 second delay when I try to get a token. If I have 2 backends then I have 6-12 seconds delay. It's very easy to test

Test connection using

for i in {1..20}; do (time keystone token-get >> log2) 2>&1 | grep real | awk '{print $2}'; done

Block one memcache backend using

iptables -I INPUT -p tcp --dport 11211 -j DROP (Simulation power outage of node)

Test the speed using

for i in {1..20}; do (time keystone token-get >> log2) 2>&1 | grep real | awk '{print $2}'; done

Also I straced keystone process with

strace -tt -s 512 -o /root/log1 -f -p PID

and got

26872 connect(9, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr("10.108.2.3")}, 16) = -1 EINPROGRESS (Operation now in progress)

though this IP is down

Also I checked the code

https://github.com/openstack/keystone/blob/master/keystone/common/kvs/core.py#L210-L237
https://github.com/openstack/keystone/blob/master/keystone/common/kvs/core.py#L285-L289
 https://github.com/openstack/keystone/blob/master/keystone/common/kvs/backends/memcached.py#L96

and was not able to find any piece of details how keystone treats with backend when it's down

There should be a logic which temporarily blocks backend when it's not accessible. After timeout period, backend should be probed (but not blocking get/set operations of current backends) and if connection is successful it should be added back to operation. Here is a sample how it could be implemented

http://dogpilecache.readthedocs.org/en/latest/usage.html#changing-backend-behavior

[Morgan Fainberg (mdrnstm)]

This behavior is the way the python memcache clients themselves work. This isn't specific to dogpile, keystone, or anything else.

The basic behavior is 'try and wait for a timeout'. Not sure what the best solution to this will be in the short-term. In the long term, the real solution will be non-persistent (no need to store) the tokens, which would eliminate the need for memcache in this regard.

[Meg McRoberts (dreidellhasa)]

Documented as "Known Issue" in 5.0.1 Release Notes

[Dolph Mathews (dolph)]

5.0.1 of what?

[Dolph Mathews (dolph)]

Based on Morgan's comment, and the fact that this is a "known issue" somewhere, it doesn't sound like there's anything for us to do in Keystone?

[Sergii Golovatiuk (sgolovatiuk)]

Currently, keystone is not ready for High availability and doesn't give any control on HA options for memcached.
I think there should be some specific values in keystone.conf where operators can tune libmemcached or pylibmc better than default settings.

[cache]
backend_behaviors=

This option should specify the behavior for backend (http://sendapatch.se/projects/pylibmc/behaviors.html#failover)

    regions.behaviors = {
        "tcp_nodelay": False,
        "ketama": True,
        "failure_limit": 2,
        "_retry_timeout": 30,
        "_auto_eject_hosts": True
    }

and toss all these settings during dogpile backend registration as specified at https://pypi.python.org/pypi/dogpile.cache

[Bogdan Dobrelya (bogdando)]

Related bug in MOS https://bugs.launchpad.net/fuel/+bug/1340657

[Alexei Kornienko (alexei-kornienko)]

I need to know type of memcached connector that is used by keystone. best option is pylibmc

[Sergii Golovatiuk (sgolovatiuk)]

currently, Fuel uses python-memcached. It lacks of HA features. pylibmc is
supported by dogpile, though pylibmc behaviors are not tunable by keystone.
*make_region*().configure() should be invoked where options are specified
for HA

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Wed, Aug 6, 2014 at 4:03 PM, Alexei Kornienko <email address hidden>
wrote:

> I need to know type of memcached connector that is used by keystone.
> best option is pylibmc
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1332058
>
> Title:
> keystone behavior when one memcache backend is down
>
> Status in OpenStack Identity (Keystone):
> Confirmed
> Status in Mirantis OpenStack:
> Confirmed
>
> Bug description:
> Hi,
>
> Our implementation uses dogpile.cache.memcached as a backend for
> tokens. Recently, I have found interesting behavior when one of
> memcache regions went down. There is a 3-6 second delay when I try to
> get a token. If I have 2 backends then I have 6-12 seconds delay. It's
> very easy to test
>
> Test connection using
>
> for i in {1..20}; do (time keystone token-get >> log2) 2>&1 | grep
> real | awk '{print $2}'; done
>
> Block one memcache backend using
>
> iptables -I INPUT -p tcp --dport 11211 -j DROP (Simulation power
> outage of node)
>
> Test the speed using
>
> for i in {1..20}; do (time keystone token-get >> log2) 2>&1 | grep
> real | awk '{print $2}'; done
>
> Also I straced keystone process with
>
> strace -tt -s 512 -o /root/log1 -f -p PID
>
> and got
>
> 26872 connect(9, {sa_family=AF_INET, sin_port=htons(11211),
> sin_addr=inet_addr("10.108.2.3")}, 16) = -1 EINPROGRESS (Operation now
> in progress)
>
> though this IP is down
>
> Also I checked the code
>
>
> https://github.com/openstack/keystone/blob/master/keystone/common/kvs/core.py#L210-L237
>
> https://github.com/openstack/keystone/blob/master/keystone/common/kvs/core.py#L285-L289
>
> https://github.com/openstack/keystone/blob/master/keystone/common/kvs/backends/memcached.py#L96
>
> and was not able to find any piece of details how keystone treats with
> backend when it's down
>
> There should be a logic which temporarily blocks backend when it's not
> accessible. After timeout period, backend should be probed (but not
> blocking get/set operations of current backends) and if connection is
> successful it should be added back to operation. Here is a sample how
> it could be implemented
>
> http://dogpilecache.readthedocs.org/en/latest/usage.html#changing-
> backend-behavior
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/keystone/+bug/1332058/+subscriptions
>

[Dolph Mathews (dolph)]

Based on comment #5 it sounds like the keystone side of this is just looking for keystone.conf [cache] backend_argument - which obviously already exists.

[Bogdan Dobrelya (bogdando)]

@Dolph, then I try to use

backend=dogpile.cache.pylibmc
and
backend_argument=behaviors:tcp_nodelay:False

I recieve an error from keystone:
ERROR: __init__() got an unexpected keyword argument 'behaviors' (HTTP 400)

[Bogdan Dobrelya (bogdando)]

Well, it could be because of old version I use (http://sendapatch.se/projects/pylibmc/index.html)...

[Bogdan Dobrelya (bogdando)]

As far as I can see Fuel puppet manifests (keystone_config) should be adjusted as well

[Vladimir Kuklin (vkuklin)]

Deployment part of this bug is tracked here:

https://bugs.launchpad.net/fuel/+bug/1340657

[Tomasz 'Zen' Napierala (tzn)]

If this is worked on, please change status to "in progress" just to keep the situation clear

[Dmitry Mescheryakov (dmitrymex)]

The bug is fixed in MOS by this commit: https://gerrit.mirantis.com/#/c/21408/8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119452

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119774

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/121166

[OpenStack Infra (hudson-openstack)]

Change abandoned by Morgan Fainberg (<email address hidden>) on branch: master
Review: https://review.openstack.org/121166
Reason: The parent needs a rebase will regenerate the config there.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119452
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0010803288748fcd3ce7dba212a54bffe7a61a0c
Submitter: Jenkins
Branch: master

commit 0010803288748fcd3ce7dba212a54bffe7a61a0c
Author: Yuriy Taraday <email address hidden>
Date: Thu Aug 28 14:27:58 2014 +0400

    Add a pool of memcached clients

    This patchset adds a pool of memcache clients. This pool allows for reuse of
    a client object, prevents too many client object from being instantiated, and
    maintains proper tracking of dead servers so as to limit delays
    when a server (or all servers) become unavailable.

    The new memcache pool backend is available either by being set as the memcache
    backend or by using keystone.token.persistence.backends.memcache_pool.Token for
    the Token memcache persistence driver.

    [memcache]
    servers = 127.0.0.1:11211
    dead_retry = 300
    socket_timeout = 3
    pool_maxsize = 10
    pool_unused_timeout = 60

    Where:
    - servers - comma-separated list of host:port pairs (was already there);
    - dead_retry - number of seconds memcached server is considered dead
      before it is tried again;
    - socket_timeout - timeout in seconds for every call to a server;
    - pool_maxsize - max total number of open connections in the pool;
    - pool_unused_timeout - number of seconds a connection is held unused in
      the pool before it is closed;

    The new memcache pool backend can be used as the driver for the Keystone
    caching layer. To use it as caching driver, set
    'keystone.cache.memcache_pool' as the value of the [cache]\backend option,
    the other options are the same as above, but with 'memcache_' prefix:

    [cache]
    backend = keystone.cache.memcache_pool
    memcache_servers = 127.0.0.1:11211
    memcache_dead_retry = 300
    memcache_socket_timeout = 3
    memcache_pool_maxsize = 10
    memcache_pool_unused_timeout = 60

    Co-Authored-By: Morgan Fainberg <email address hidden>
    Closes-bug: #1332058
    Closes-bug: #1360446
    Change-Id: I3544894482b30a47fcd4fac8948d03136fd83f14

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119774
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=045cddcea2ecefccecbb40d4249b915c3f1faae3
Submitter: Jenkins
Branch: master

commit 045cddcea2ecefccecbb40d4249b915c3f1faae3
Author: Morgan Fainberg <email address hidden>
Date: Sun Sep 21 13:20:35 2014 -0700

    Add an optional advanced pool of memcached clients

    This patchset adds an advanced eventlet safe pool of memcache clients. This
    allows the deployer to configure auth_token middleware to utilize the new
    pool by simply setting 'memcache_use_advanced_pool' to true. Optional
    tunables for the memcache pool have also been added.

    Co-Authored-By: Morgan Fainberg <email address hidden>
    Closes-bug: #1332058
    Closes-bug: #1360446
    Change-Id: I08082b46ce692cf4df449d48dac94718f1e98a6c

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.1, package release == fuel5.0.3.mira8.git.de075cb.7361afa

Changeset: https://review.fuel-infra.org/561
project: openstack/keystone
branch: openstack-ci/fuel-5.0.3/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Update a pool of memcached clients from upstream
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.1-fuel5.0.3.mira8.git.de075cb.7361afa.noarch.rpm
openstack-keystone-doc-2014.1.1-fuel5.0.3.mira8.git.de075cb.7361afa.noarch.rpm
python-keystone-2014.1.1-fuel5.0.3.mira8.git.de075cb.7361afa.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable-561/centos

[OSCI Robot (oscirobot)]

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.1, package release == fuel5.0.3~mira8+git.de075cb.7361afa

Changeset: https://review.fuel-infra.org/561
project: openstack/keystone
branch: openstack-ci/fuel-5.0.3/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Update a pool of memcached clients from upstream
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.1-fuel5.0.3~mira8+git.de075cb.7361afa_all.deb
keystone_2014.1.1-fuel5.0.3~mira8+git.de075cb.7361afa_all.deb
python-keystone_2014.1.1-fuel5.0.3~mira8+git.de075cb.7361afa_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.3-stable-561/ubuntu

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737579
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=bb0393623ca8687714342d2b0cc73cc6c126ecde
Submitter: Zuul
Branch: master

commit bb0393623ca8687714342d2b0cc73cc6c126ecde
Author: Lance Bragstad <email address hidden>
Date: Tue Jun 23 11:37:06 2020 -0500

    Write a symptom for checking memcache connections

    This makes it easier for operators to troubleshoot connection issues to
    Memcached.

    Related-Bug: 1332058

    Change-Id: I6e67363822480314b93608bb1eae3514f1480f6d