Some haproxy backend has down state during mysql one-by-one termination

[Alexandr Kostrikov]

During swarm[0] run there were failure on scenario:
        Scenario:
            1. Terminate mysql
            2. Wait while it is being restarted
            3. Verify it is restarted
            4. Go to another controller
            5. Run OSTF

fuel_health.tests.ha.test_haproxy: DEBUG: Dead backends ['mysqld node-1 Status: DOWN 1/3/L7OSessions: 0 Rate: 0 ']

[0] https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/167/testReport/(root)/ha_neutron_mysql_termination/ha_neutron_mysql_termination/

Additional information:
There is no env available, need to wait for reproduce

[Oleksiy Molchanov (omolchanov)]

2017-01-04 05:07:11 FAILURE Check state of haproxy backends on controllers (fuel_health.tests.ha.test_haproxy.HAProxyCheck.test_001_check_state_of_backends) Some haproxy backend has down state..

Jan 04 05:07:10 [4897] node-1.test.domain.local pengine: info: clone_print: Clone Set: clone_p_mysqld [p_mysqld]
Jan 04 05:07:10 [4897] node-1.test.domain.local pengine: info: short_print: Started: [ node-1.test.domain.local node-4.test.domain.local node-5.test.domain.local ]

[Oleksiy Molchanov (omolchanov)]

http://paste.openstack.org/show/594036/ - haproxy logs

[Vladimir Kuklin (vkuklin)]

So, as we can see from the logs, haproxy backends get up a little bit later because haproxy has to perform at least 3 checks with interval of 3 seconds before declaring backends online. This means that we need to introduce a loop in the tests which checks several times which backends are online with an interval of 9 seconds.

[Vladimir Kuklin (vkuklin)]

2017-01-04 05:07:11 FAILURE Check state of haproxy backends on controllers (fuel_health.tests.ha.test_haproxy.HAProxyCheck.test_001_check_state_of_backends) Some haproxy backend has down state.. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor

<133>Jan 4 05:07:14 node-1 haproxy[19709]: Server mysqld/node-1 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 81ms. 1 active and 2 backup servers online. 0 sessions requeued, 0 total in queue.

[Vladimir Kuklin (vkuklin)]

I mean that we need to wait for pacemaker to bring mysql up and then let haproxy set backends as online.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: stable/mitaka
Change author: Vladimir Jigulin <email address hidden>
Review: https://review.fuel-infra.org/37244

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/37244
Submitter: Dmitry Burmistrov <email address hidden>
Branch: stable/mitaka

Commit: 364972dc45a91db77abde093e6a8972ef14e4c3c
Author: Vladimir Jigulin <email address hidden>
Date: Wed Nov 22 08:42:03 2017

Wait for haproxy status update in haproxy/mysql-termination tests

Change-Id: Ie42e9ea5d516213517f5050bd14b9bd0376b8c7e
Closes-Bug: #1654276