haproxy-status.sh exit status: 1, Error: 2016/10/10 14:06:00 socat[20638] E connect(5, AF=1 "/var/lib/haproxy/stats", 24): No such file or directory

[Alexandra]

Swarm test is failed on 10.0 #839
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.controller_replacement/146/testReport/(root)/deploy_ha_neutron_vlan_ctrl_replacement/

Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check state of haproxy backends on controllers (failure) Can't get state of backends. Please refer to OpenStack logs for more details.

Extraction from ostf.log:
2016-10-10 14:06:00 ERROR (test_mixins) Command 'haproxy-status.sh | egrep -v "BACKEND|FRONTEND"', exit status: 1, Error:
2016/10/10 14:06:00 socat[20638] E connect(5, AF=1 "/var/lib/haproxy/stats", 24): No such file or directory
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 176, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/ha/test_haproxy.py", line 60, in _check_haproxy_backend
    return remote.exec_command("{0}{1}".format(cmd, ''.join(grep)))
  File "/usr/lib/python2.7/site-packages/fuel_health/common/ssh.py", line 165, in exec_command
    strerror=''.join(err_data).join(out_data))
SSHExecCommandFailed: Command 'haproxy-status.sh | egrep -v "BACKEND|FRONTEND"', exit status: 1, Error:
2016/10/10 14:06:00 socat[20638] E connect(5, AF=1 "/var/lib/haproxy/stats", 24): No such file or directory

[Alexandra (aallakhverdieva)]

One more failed test:
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.controller_replacement/146/testReport/(root)/deploy_ha_neutron_tun_ctrl_replacement/

[Stanislaw Bogatkin (sbogatkin)]

I can't reproduce this bug on local env. But when looked at system tests, we can see that there are too much memory and cpu consumed at the error time. So it can be a cpu/mem resources problem.

[Alexey Shtokolov (ashtokolov)]

This test works for 9.2 https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.controller_replacement/

[Dmitry Ilyin (idv1985)]

This script communicates with the haproxy daemon via the unix socket which means that haproxy should be running on the same system the script is being run.

Perhaps, there should be a better error message or the script should support connection to the haproxy via virtual ip?

Or haproxy is just not running for some reason.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/419472

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/419472
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.