Destroy haproxy test failed with waiting timeout while haproxy recovers

[Tatyanka]

https://172.18.160.103/job/9.0.system_test.ubuntu.ha_neutron_destructive/65/testReport/%28root%29/ha_neutron_haproxy_termination/ha_neutron_haproxy_termination/

Error Message

Waiting timed out

Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/helpers/decorators.py", line 119, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover.py", line 137, in ha_neutron_haproxy_termination
    super(self.__class__, self).ha_haproxy_termination()
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover_base.py", line 402, in ha_haproxy_termination
    wait(haproxy_started, timeout=20)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 96, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Waiting timed out

1. Error message wit some details about waiting timeout is missed
2. After revert - all haproxy was restored

[Maksym Strukov (unbelll)]

Env: 9.0 iso #197 https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_destructive/78/testReport/(root)/ha_neutron_haproxy_termination/

After revert haproxy is down, but I can start it manually

```
root@node-3:~# echo $?
1

root@node-3:~# service haproxy status
haproxy not running.

root@node-3:~# service haproxy start
 * Starting haproxy haproxy [ OK ]
root@node-3:~# [ -f /var/run/haproxy.pid ] && [ "$(ps -p $(cat /var/run/haproxy.pid) -o pid=)" == "$(pidof haproxy)" ]
root@node-3:~# echo $?
1
root@node-3:~# echo $?
0
root@node-3:~# echo $?
0
```

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/308161

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308787

[OpenStack Infra (hudson-openstack)]

Change abandoned by Volodymyr Shypyguzov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/308787

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308161
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=f9f7cef2572259c0cc0a310d23eafa9aa29e8868
Submitter: Jenkins
Branch: master

commit f9f7cef2572259c0cc0a310d23eafa9aa29e8868
Author: Volodymyr Shypyguzov <email address hidden>
Date: Wed Apr 20 10:11:03 2016 +0300

    Fix haproxy check commands

    +Change command to point at correct pid file location
    +Refactor code in favor to ssh_manager

    Change-Id: I3968d49e017649a9efad5e5c5de236afe1a5b137
    Closes-Bug: #1564903
    Implements blueprint sshmanager-integration

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308787
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=b8772d79ef802ba0f0742f4bdd0ed9ff03327d4b
Submitter: Jenkins
Branch: stable/mitaka

commit b8772d79ef802ba0f0742f4bdd0ed9ff03327d4b
Author: Volodymyr Shypyguzov <email address hidden>
Date: Wed Apr 20 10:11:03 2016 +0300

    Fix haproxy check commands

    +Change command to point at correct pid file location
    +Refactor code in favor to ssh_manager

    Change-Id: I3968d49e017649a9efad5e5c5de236afe1a5b137
    Closes-Bug: #1564903
    Implements blueprint sshmanager-integration
    (cherry picked from commit f9f7cef2572259c0cc0a310d23eafa9aa29e8868)

[Tatyanka (tatyana-leontovich)]

verified on run for 285 iso