[10.0][swarm] Task restart-haproxy failed

[Dmitry Belyaninov]

There are few swarm tests failed with error:

Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[restart-haproxy/3] Stopping the deployment process!'

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.numa_cpu_pinning/157/testReport/(root)/change_pinned_cpu_and_redeploy/change_pinned_cpu_and_redeploy/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.services_reconfiguration_thread_2/169/testReport/(root)/upload_config_for_node_and_env_in_transitional_state/upload_config_for_node_and_env_in_transitional_state/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.install_plugin_after_cluster_create/147/testReport/(root)/five_ctrl_enable_installed_after_deploy_add/five_ctrl_enable_installed_after_deploy_add/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.thread_2_separate_services/172/testReport/(root)/separate_rabbit_service_add_delete_node/separate_rabbit_service_add_delete_node/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.controller_replacement/169/testReport/(root)/deploy_ha_neutron_tun_ctrl_replacement/deploy_ha_neutron_tun_ctrl_replacement/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ha_neutron_tun_scale/168/testReport/(root)/neutron_tun_scalability/neutron_tun_scalability/

[Oleksiy Molchanov (omolchanov)]

I have checked

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ha_neutron_tun_scale/168/testReport/(root)/neutron_tun_scalability/neutron_tun_scalability/

And puppet.log ends with

2017-02-01 23:27:00 +0000 Scope(Class[Osnailyfacter::Cluster_haproxy::Restart_haproxy]) (notice): MODULAR: cluster_haproxy/restart_haproxy.pp
2017-02-01 23:27:00 +0000 Puppet (notice): Compiled catalog for node-2.test.domain.local in environment production in 0.20 seconds
2017-02-01 23:27:02 +0000 Puppet (notice): Haproxy service will be restarted
2017-02-01 23:27:02 +0000 /Stage[main]/Osnailyfacter::Cluster_haproxy::Restart_haproxy/Notify[Haproxy service will be restarted]/message (notice): defined 'message' as 'Haproxy service will be restarted'

But in logs/messages I can see that node was alive.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/429786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/436514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/438911

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/436514
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6ceca619338508479df2a67f33af20353bcb0a4a
Submitter: Jenkins
Branch: master

commit 6ceca619338508479df2a67f33af20353bcb0a4a
Author: Dmitry Ilyin <email address hidden>
Date: Tue Feb 21 18:59:01 2017 +0300

    Revert: Scale controllers up/down using pacemaker m-mode

    Revert adding the cluster_mm and cluster_mm_off tasks
    which turn the maintenance mode on the cluster nodes.

    Change-Id: I6fe7b24d5fb4610ae33ecaabfb6648efabc32dfc
    Releted-Bug: 1661177

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438911
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a87e9264904fbb6111cd137e820cfb935055ec7f
Submitter: Jenkins
Branch: stable/newton

commit a87e9264904fbb6111cd137e820cfb935055ec7f
Author: Dmitry Ilyin <email address hidden>
Date: Tue Feb 21 18:59:01 2017 +0300

    Revert: Scale controllers up/down using pacemaker m-mode

    Revert adding the cluster_mm and cluster_mm_off tasks
    which turn the maintenance mode on the cluster nodes.

    Change-Id: I6fe7b24d5fb4610ae33ecaabfb6648efabc32dfc
    Releted-Bug: 1661177

[Dmitry Pyzhov (dpyzhov)]

Revert should help. Closing the bug as Fix Committed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/442436

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/442436
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fede1d12b2bc1c3ef59502f47b2335dd465fc079
Submitter: Jenkins
Branch: stable/ocata

commit fede1d12b2bc1c3ef59502f47b2335dd465fc079
Author: Dmitry Ilyin <email address hidden>
Date: Tue Feb 21 18:59:01 2017 +0300

    Revert: Scale controllers up/down using pacemaker m-mode

    Revert adding the cluster_mm and cluster_mm_off tasks
    which turn the maintenance mode on the cluster nodes.

    Change-Id: I6fe7b24d5fb4610ae33ecaabfb6648efabc32dfc
    Releted-Bug: 1661177
    (cherry picked from commit 6ceca619338508479df2a67f33af20353bcb0a4a)

[Igor Gajsin (igajsin)]

The given fix removes the tasks that not present in the stable/mitaka. So it's invalid for this branch.

[Vladimir Khlyunev (vkhlyunev)]

Still reproduces - https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.install_plugin_after_cluster_create/209/testReport/
ISO 1561

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/429786
Reason: This repo is retired now, no further work will get merged.