Keystone fails over slowly

[Dmitry Mescheryakov]

Steps to reproduce

1. Install HA env with 3 controllers and 1 compute node
2. Shut down one of the controllers
3. Get a new token with 'keystone token-get'
4. Do 'keystone tenant-list' using token from step #4
5. Repeat steps 3-4 several times

With some probability, step 4 (keystone tenant-list) will fail with message 'Invalid OpenStack Identity credentials'. You might need to get up and shut down controller several times before you get a repro. But once you get it, many new tokens generated in step #4 will fail on step #4. The behaviour persists until the controller is up.

[Boris Bobrov (bbobrov)]

Changed the description a bit.

The bug should be fixed by https://review.fuel-infra.org/#/c/7407/. With the fix, one should wait TCP_KEEPALIVE seconds after node failure. The value (TCP_KEEPALIVE) is out of keystone's control, someone who deals with network configuration should have a look here.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/7407
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 50c5e04e34ec025457f1ceb73763b9f4d3de1798
Author: Boris Bobrov <email address hidden>
Date: Thu Jun 4 11:04:50 2015

add HA memcache client

Related-Bug: 1446590
Closes-Bug: 1461655
Change-Id: I0d1cec45c135cd102541ab8ab86af6ca4d6debcb

[Dmitry Mescheryakov (dmitrymex)]

Here is the fix for the bug: https://review.fuel-infra.org/#/c/7504/

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/8188

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8188

[Fuel Devops McRobotson (fuel-devops-robot)]

Change restored by Boris Bobrov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8188