We need to wait for keystone and memchached become operatable in all ha destructive tests

[Tatyanka]

In each tests were revert take place we need to waiting for memchached become operable (near 4 minutes) otherwise we Will fail on ostf with 401 error from keystone or 500 error with message pool is full
like here http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.centos.thread_5/99/testReport/%28root%29/ha_nova_haproxy_termination/
and in other destructive tests

[Mike Scherbakov (mihgen)]

How destructive tests are related to this? If we shut down controller, and then have to wait for 4 minutes, then it's HA story. If it's just about reverting all controllers from snapshots, and then waiting for 4 min before it all becomes operational, then I would agree that we should just wait for 4 minutes.
Though ideally we would want to decrease the time anyway in order to speed up our testing.

[Tatyanka (tatyana-leontovich)]

@Mike it is related to revert snapshot before destructive actions

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/187925

[Boris Bobrov (bbobrov)]

Folks, with patch https://review.fuel-infra.org/#/c/7407/ wait time is TCP_KEEPALIVE seconds. The value is about 60 seconds.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/7407
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 50c5e04e34ec025457f1ceb73763b9f4d3de1798
Author: Boris Bobrov <email address hidden>
Date: Thu Jun 4 11:04:50 2015

add HA memcache client

Related-Bug: 1446590
Closes-Bug: 1461655
Change-Id: I0d1cec45c135cd102541ab8ab86af6ca4d6debcb

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/187925
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=c75a695639df98d6be85ce129c0fb1f63e685e42
Submitter: Jenkins
Branch: master

commit c75a695639df98d6be85ce129c0fb1f63e685e42
Author: Artem Roma <email address hidden>
Date: Wed Jun 3 12:36:37 2015 +0300

    Add waiting for keystone to be fully available after haproxy restart

    Sometimes keystone is not available right after haproxy restart. To
    consider that in the tests preliminary check of the situation added in
    form of ostf sanity set execution.

    Change-Id: I4a5d6adcc13ca1e996e0ed6bdbaa0ef35352f4b9
    Partial-Bug: #1446590

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/8188

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8188

[Fuel Devops McRobotson (fuel-devops-robot)]

Change restored by Boris Bobrov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8188