[library] 'keystone' delays lead to unstable MOS operations

[Dennis Dmitriev]

Upstream bug: https://bugs.launchpad.net/keystone/+bug/1332058

Confirmed for OpenStack HA environment.

When several OS controller nodes are used with 'memcached' installed on each of them, each 'keystone' instance is configured to use all of the 'memcached' instances.
Thus, if one of the controller nodes became inaccessible, then whole cluster may cease to be workable because of a lot of delays in the memcached backend.

There is no acceptable workaround at the moment which would allow the use all of available 'memcached' instances without such kind of issue.

[Dennis Dmitriev (ddmitriev)]

May be similar to bug: https://bugs.launchpad.net/fuel/+bug/1337923

Steps to reproduce:

Steps to Reproduce:
1. Create HA cluster with CentOS, neutron vlan
2. Add 3 nodes with controller role
3. Add 2 nodes with compute role
4. Deploy the cluster
5. Run network verification
6. Run OSTF
7. turn of primarry controller
8. wait for about 5 minutes
9. Run OSTF

Expected results:
Only "Check that required services are running" failed,
All "Functional tests" and "HA tests" passed.

Actual results:
"Functional tests" that launch an instance are failed with "Timed out waiting to become ACTIVE..." errors.

To make sure that this OSTF behaviour is affected by 'keystone':

1. On the rest operational controllers remove IP of the primary controller (it is 192.168.10.3 in the following example) from the IPs lists in the 'keystone' config file /etc/keystone/keystone.conf:
...
[cache]
backend_argument=url:192.168.10.3,192.168.10.4,192.168.10.5
...
[memcache]
servers=192.168.10.3:11221,192.168.10.4:11221,192.168.10.5:11221
..

2. restart openstack-keystone service on all controllers
3. run OSTF

[Yuriy Taraday (yorik-sar)]

This issue represents the problem in using cache as a storage: You shouldn't do that.

There are two places where Keystone can use Memcached: as a storage for tokens and as a cache for everything.

For caching one shouldn't use all memcached instances: I'd be more optimal to use only one instance that resides on the same controller. That would make controller less dependent from neighbor controllers and remove unnecessary network interactions.

For tokens one should just use PKI that doesn't use storage backend constantly. With introduction of revocation events persistent storage for them became obsolete.

[Sergii Golovatiuk (sgolovatiuk)]

Fuel configures memcached as a storage for keystone tokens only. The problem is when you have several memcache servers and use roundrobin mechanisms for API memcache client should iterate over all memcache backends to be able to find a key-value pair.

[Yuriy Taraday (yorik-sar)]

As I said, you should use PKI instead.

[Bogdan Dobrelya (bogdando)]

Perhaps we should try PKI mode once puppet-keystone upstream sync patches merged

[Bogdan Dobrelya (bogdando)]

For PKI tokens case we could as well consider putting memcached servers under HAproxy and configure LB by specifying a single backend_argument then. Makes sense?

[Nastya Urlapova (aurlapova)]

Bogdan, please add action items for repair this issue.

[Bogdan Dobrelya (bogdando)]

Action items could be as the following:
* Merge keystone upstream manifests once other dependent modules (neutron, etc) synced as well
OR just use the keystone module we have in library now
* Configure keystone to be deployed with PKI mode instead of UUID (there are could be much better chances to do so using upstream manifests, though)
* Try to reproduce bug again

[Bogdan Dobrelya (bogdando)]

Related bug in keystone https://bugs.launchpad.net/mos/+bug/1332058

[Sergii Golovatiuk (sgolovatiuk)]

According to latest tests, this bug doesn't affect FUEL anymore

[Sergii Golovatiuk (sgolovatiuk)]

In order to fix the issue we should add libmemcached and python-pylibmc packages

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112829

[Artem Panchenko (apanchenko-8)]

api: '1.0'
astute_sha: b52910642d6de941444901b0f20e95ebbcb2b2e9
auth_required: false
build_id: 2014-08-10_02-01-17
build_number: '420'
feature_groups:
- mirantis
fuellib_sha: d699fc178559e98cfd7d53b58478b46553ffe39e
fuelmain_sha: 9d4463400b4924159c978af43855e48bcf2a84b2
nailgun_sha: 2741cdc0f0615263db2f176899d406207ec4ac04
ostf_sha: acf52a59e04fa74d2ed2b68ea225f4d24403b264
production: docker
release: '5.1'

Can't confirm that this issue doesn't affect Fuel anymore. Just tested keystone performance on bare metal and got the following results, when all controllers are online:

[root@node-6 ~]# time for i in {1..10}; do keystone user-list &>/dev/null; done

real 0m6.785s
user 0m3.902s
sys 0m0.513s

and after primary controller shutdown:

[root@node-6 ~]# time for i in {1..10}; do keystone user-list &>/dev/null; done

real 1m31.950s
user 0m3.884s
sys 0m0.524s

As you can see the difference is huge and that's why all api calls become very slowly if one of controllers is down. When I redirect the packets sent to the inaccessible memcached instance to the localhost on running controllers (iptables -t nat -I OUTPUT -d 192.168.0.3 -p tcp --dport 11211 -j DNAT --to 127.0.0.1:11211) everything become fine.

Sergii, JFYI I've tried to use dogpile.cache.pylibmc driver for keystone (manually installed pylibmc 1.3.0 and latest libmemcached on controllers) and in case of primary controller (192.168.0.3) is down I always get the following error from keystone:

http://paste.openstack.org/show/92992/

 I also tried to use bmemcached as backend, but then it always returns 504 error if old primary controller is down:

http://paste.openstack.org/show/92994/

[Sergii Golovatiuk (sgolovatiuk)]

Artem,

I have the identical results on my environment. Unfortunately, there is no easy fix. MOS Keystone team is working to specify backend behavior for pylibmc. It will allow Fuel operators to specify behavior more optimal than default one.

[Dmitry Borodaenko (angdraug)]

I marked https://bugs.launchpad.net/mos/+bug/1353419 as duplicate of this bug, I think this bug should also be marked as affecting mos.

[Dmitry Mescheryakov (dmitrymex)]

Dmitry, on MOS side this bug is tracked in https://bugs.launchpad.net/keystone/+bug/1332058 I know that we decided not to use upstream bugs for MOS+Fuel dev, but that one was created before our decision took effect and we keep using it.

[XiaBing Yao (yao3690093-o)]

use haproxy can solve this problem.

this is my haproxy.cfg about memcache server
listen memcache
    bind controller:11221
    balance source
    mode tcp
    server controller1 controller1:11211 check inter 2000 rise 2 fall 3
    server controller2 controller2:11211 check inter 2000 rise 2 fall 3 backup

vim /etc/keystone/keystone.conf
[memcache]
servers = controller:11221

vim /etc/nova/nova.conf
[DEFAULT]
memcached_servers = controller:11221

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/112829

[Aleksandr Shaposhnikov (alashai8)]

Guys, I think that here is the issue of all problem in 5.0 and later releases of Fuel.
https://bugs.launchpad.net/keystone/+bug/1360446

And regarding memcached under HA. It wouldn't work under LB because memcached servers didn't synchronized between each other and have completely different data in each and every server. Here is comply of ways of solving that:
1. Spawn memcached using corosync in haproxy/VIP addresses namespaces on only one node. Using VIP MGMT to access haproxy.
2. I hate using of haproxy behind of memcached but the only way (As I see) is to have memcached under haproxy is to have one ACTIVE memcached abd and remaining in BACKUP/PASSIVE mode.

[Sergii Golovatiuk (sgolovatiuk)]

VIP controlled by corosync for memcached is something we want to avoid. There are many problems especially with horizontal scalability as we'll be limited to one instance of memcached. We are trying to use pylibmc/libmemcached right now which will allow us to detect "dead" memcached servers, spread load across all memcached server, have a good horizontal scalability distributing load across all controllers.

[Meg McRoberts (dreidellhasa)]

What is the appropriate text for the Release Notes? The current draft is in https://review.openstack.org/#/c/117338/3/pages/release-notes/v5-1/038-resolved-ha-issues.rst -- what I have for this issue is clearly not correct.

[Dmitry Mescheryakov (dmitrymex)]

Meg, the fix for the bug is in progress currently. If we fully fix it, we will not need to put info about it in release notes for 5.1/5.0.2. For your reference, the same bug on MOS side is tracked here: https://bugs.launchpad.net/keystone/+bug/1332058

[Yuriy Taraday (yorik-sar)]

I've prepared a patch in Keystone that should also fix this problem: https://gerrit.mirantis.com/21408 .
This patch requires some changes in config (only changed/added config values presented):

[cache]
backend=keystone.cache.memcache_pool # turn on new cache backend
backend_argument=pool_maxsize:100 # add this one, keep "backend_argument=url:<...>" there

Other options might be needed in future as well (see commit message for details), but these two should be enough.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/118382

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/118382
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=203ef3179007cffe3236032e61ecbaf1cd20605f
Submitter: Jenkins
Branch: master

commit 203ef3179007cffe3236032e61ecbaf1cd20605f
Author: Vladimir Kuklin <email address hidden>
Date: Tue Sep 2 19:30:36 2014 +0400

    Change keystone memcache backend

    Use keystone memcache pool backend
    in order to fix scalability and
    failover problems

    Also fixing file_line "match" + "after" insertion logic.

    Change-Id: Idfe4b54caa0d96a93e93bfff12d8b6216f83e2f1
    Closes-bug: #1364401
    Closes-bug: #1340657

[Dmitry Mescheryakov (dmitrymex)]

Now that we've ported Yuriy's fix to 5.0.2, we need to adjust 5.0.2 puppets as well

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/122230

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/122230

[Sergii Golovatiuk (sgolovatiuk)]

This change requires keystone package from 5.1 where Yuri Taraday added a new eventlet safe memcache driver for keystone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135629

[Aleksandr Didenko (adidenko)]

OK, I've checked 5.1.1. It still has older version of keystone.cache.memcache_pool with old arguments (backend and backend_argument). So no changes needed for 5.1.1 and fix is already released in 5.1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135629
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4900833f4d355fe729f20cc57aef341b4fb248c2
Submitter: Jenkins
Branch: master

commit 4900833f4d355fe729f20cc57aef341b4fb248c2
Author: Aleksandr Didenko <email address hidden>
Date: Wed Nov 19 17:32:00 2014 +0200

    Restore keystone cache.memcache_pool backend

    Restore lost change https://review.openstack.org/#/c/118382/
    Plus refactor to setup valid keystone.cache.memcache_pool params.

    Closes-bug: #1340657
    Change-Id: Id30ad74afe8cef17a6c42f6dfc5c9c581348bc1f

[Anastasia Palkina (apalkina)]

Verified on ISO #49

"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/143084

[Artem Panchenko (apanchenko-8)]

Guys, seems the problem still exists, I reproduced this issue on bare metal lab. Here are my steps:

1. Create HA environment (Ubuntu+NeutronVlan+Ceph+ceilometer).
2. Add 3 controller+mongo and 4 copmute+ceph nodes.
3. Deploy changes. Run OSTF. All tests passed.
4. Shutdown some controller (or just block traffic to memcached that is running on it - `iptables -I INPUT -p tcp --dport 11211 -j DROP`)

Expected result:

 - OS services continue to work fine, environment passes health checks

Actual result:

 - lots of OSTF smoke tests fails by timeout "AssertionError: Failed to get to expected status. In error state."

When all (3) memcached instances are available keystone needs 0.1-0.2 sec to create new token:

http://paste.openstack.org/show/154430/

and opening of '/horizon/admin/networks/' URL in browser takes 2-3 seconds.

After destroying one of controllers keystone spends >3 seconds creating a token:

http://paste.openstack.org/show/154431/

Loading of '/horizon/admin/networks/' URL in browser takes 10-15 seconds.

Fuel version( 6.0 #56): http://paste.openstack.org/show/154432/

Unfortunately I can't provide diagnostic snapshot now because it's too huge, but I will try to reproduce the issue on another fresh environment and attach logs.

[Nastya Urlapova (aurlapova)]

@Artem please create other issue, looks like we have incorrect keystone behaviour here and discuss it with Vova K. he could provide more information.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/143084