[Keystone] Unable to get a connection from pool id 140696440754064 after 10 seconds

[Leontii Istomin]

api: '1.0'
astute_sha: 055b2d82fe8499b27c7047295e2e36a7a2c5d430
auth_required: true
build_id: 2015-04-16_21-30-10
build_number: '317'
feature_groups:
- mirantis
fuellib_sha: db5f39e96e7ab9f79691202755e547bf8242661f
fuelmain_sha: 0de2d2039e76839d339f977df45b111bef7200d6
nailgun_sha: 52d92c86e68602fb5dd2f3b8870a773d20a387ef
openstack_version: 2014.2-6.1
ostf_sha: b0991dbad159f53d335efa5d31cb94016ad5312e
production: docker
python-fuelclient_sha: 279ffc358e40dbdc162cfe76898cbd0874529f1f
release: '6.1'

Successfully deployed the following configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-vlan,Ceph-all,Nova-debug,Nova-quotas, 6.1-317
Controllers:3 Computes:47

But during rally get_token test we have faced with this issue:
http://paste.openstack.org/show/222835/

it happened only on one controller node

Diagnostic snapshot is here http://mos-scale-share.mirantis.com/fuel-snapshot-2015-05-14_08-14-00.tar.xz

[Dmitry Mescheryakov (dmitrymex)]

Leontiy, did any test failed, or these were just errors in log? Right now it looks like the later and so I am setting bug priority to low as just errors in log don't bite users. Please comment if it was otherwise, we will reevaluate the bug then.

[Leontii Istomin (listomin)]

get token from keystone rally test was failed with this error:
http://paste.openstack.org/show/223752/
sorry that I didn't explain it earlier.

[Boris Bobrov (bbobrov)]

Leontiy, this bug is extremely hard to debug because it happened only on one node. I checked config and logs and could not find anywhere to start debugging from. Could you please try reproducing it and note the order or the steps you took to reproduce it?

[Leontii Istomin (listomin)]

I happened during rally test. You can try to reproduce the issue with 94 requesets per second to get token. It should be failed after 150000 request

[Boris Bobrov (bbobrov)]

> You can try to reproduce the issue with 94 requesets per second to get token. It should be failed after 150000 request

So, you were able to reproduce it, right?

[Leontii Istomin (listomin)]

The issue has been reproduced with following configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-gre,Ceph-all,Nova-debug,Nova-quotas, 6.1-317
Controllers:3 Computes:47

Rally report is attached.
Diagnostic Snapshot is here: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-05-20_09-38-14.tar.xz

[Alexander Makarov (amakarov)]

It is tested against MOS 6.1 env.
MOS 7.0 wil have non-persistent tokens so the issue will be no more.

[Leontii Istomin (listomin)]

hasn't been reproduced since 6.1-425

[Dina Belova (dbelova)]

We have reproduced it on #511 ISO during rally testing

[Leontii Istomin (listomin)]

reproduced the issue with the following configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-gre,Ceph-all,Nova-debug,Nova-quotas, 6.1-511
Controllers:3 Computes:47

api: '1.0'
astute_sha: 7766818f079881e2dbeedb34e1f67e517ed7d479
auth_required: true
build_id: 2015-06-05_17-16-32
build_number: '511'
feature_groups:
- mirantis
fuel-library_sha: f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d
fuel-ostf_sha: 7c938648a246e0311d05e2372ff43ef1eb2e2761
fuelmain_sha: bcc909ffc5dd5156ba54cae348b6a07c1b607b24
nailgun_sha: 4340d55c19029394cd5610b0e0f56d6cb8cb661b
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'

Diagnostic Snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2015-06-08_15-41-56.tar.xz

[Bogdan Dobrelya (bogdando)]

Note, we have a HCF for the 6.1, so this bug should go to the 6.1.x milestone when it will be added

[Bogdan Dobrelya (bogdando)]

(MOS 7.0 wil have non-persistent tokens so the issue will be no more.)

[Timur Nurlygayanov (tnurlygayanov)]

As we discussed, we can fix the issue in MOS 6.1 and we also should commit fix in master branch too to have it in the next releases.

[Max Yatsenko (myatsenko)]

To fix the bug for 6.1 a request was prepared for upstream - https://review.openstack.org/#/c/189811/ :

"Set new parameter 'pool_maxsize' for memcache. Default value was set to 100 to fix an issue with getting token from keystone after 10 sec timeout."

[Vladimir Kuklin (vkuklin)]

We must fix it for 7.0 also until we implement non-persistent tokens. Let's not hijack our development flow.

[Max Yatsenko (myatsenko)]

Change request for 7.0 - https://review.openstack.org/#/c/190632/