OSTF test "Check state of haproxy backends on controllers" failed after deploy

[Sergey Novikov]

Detailed bug description:
OSTF test "Check state of haproxy backends on controllers" failed after reinstallation of controller. Issue was caught by system tests
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.error_node_reinstallation/110/testReport/(root)/reinstall_failed_primary_controller_deployment/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.error_node_reinstallation/110/testReport/(root)/reinstall_failed_regular_controller_deployment/

Steps to reproduce:
 please take a look at corresponding system tests

Expected results:
 OSTF tests passed

Actual result:
 test "Check state of haproxy backends on controllers" failed

Description of the environment:
MOS 9.0 ISO #364

Additional information:
 Haproxy status http://paste.openstack.org/show/497321/

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Sergey Novikov (snovikov)]

Seems, other two system tests are affected by this issue
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.jumbo_frames/111/testReport/(root)/jumbo_frames_neutron_diff_net_vlan/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.jumbo_frames/111/testReport/(root)/jumbo_frames_neutron_vxlan/

[Dmitriy Kruglov (dkruglov)]

Another occurrence on the latest 9.0 swarm run:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.cgroup_ha/49/testReport/(root)/deploy_ha_cgroup/deploy_ha_cgroup/

[Timur Nurlygayanov (tnurlygayanov)]

Reproduced on SWARM tests for Murano as well:

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.services_ha/109/testReport/(root)/deploy_murano_ha_with_tun/deploy_murano_ha_with_tun/

[Ksenia Svechnikova (kdemina)]

Reproduced on swarm

Env: 9.0 mos iso #372
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.numa_cpu_pinning/29/testReport/(root)/cpu_pinning_on_two_compute/cpu_pinning_on_two_compute/

Steps:
        1. Revert snapshot "basic_env_for_numa_cpu_pinning"
        2. Pin maximum CPU for the nova on the first compute
        3. Pin minimun CPU for the nova on the second compute
        4. Verify setting was successfully applied
        5. Deploy cluster
        6. Check new filters are enabled in nova.conf at controller
        7. Check nova.conf contains pinned CPU at compute
        8. Run OSTF

[Timur Nurlygayanov (tnurlygayanov)]

Reproduced on 10.0 ostf gates as well:

https://ci.fuel-infra.org/job/master.fuel-ostf.pkgs.ubuntu.gate_ostf_update/412/console

[Michael Polenchuk (mpolenchuk)]

message Deployment has failed. All nodes are finished. Failed tasks: Task[hiera/id] Stopping the deployment process!