[swarm] Nodes are offline after redeploy

[Dmitry Belyaninov]

Detailed bug description:
There is failed swarm test:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.unlock_settings_tab_states/131/testReport/(root)/unlock_settings_tab_positive/

AssertionError: Task 'deploy' has incorrect status. error != ready, 'Nodes "slave-04_compute (id=4, mac=64:9c:58:ca:de:d5),slave-02_controller (id=3, mac=64:ee:99:83:0b:9b),slave-01_controller (id=1, mac=64:09:e9:32:b2:91),slave-05_compute (id=5, mac=64:db:38:8b:ab:3b),slave-03_controller (id=2, mac=64:f3:3e:a7:68:e4)" are offline. Remove them from environment and try again.'

Steps to reproduce:
run the unlock_settings_tab_positive fuel-qa test
Expected results:
pass
Actual result:
fail
Reproducibility:
yes
Workaround:

Impact:
 <put your information here>
Description of the environment:
 Operation system: <put your information here>
 Versions of components: <put your information here>
 Reference architecture: <put your information here>
 Network model: <put your information here>
 Related projects installed: <put your information here>
Additional information:
 <put your information here>

[Oleksiy Molchanov (omolchanov)]

CI team, can you check what have happened?

[ElenaRossokhina (esolomina)]

New failure https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.unlock_settings_tab_states/152/testReport/(root)/unlock_settings_tab_positive/unlock_settings_tab_positive/

[Pawel Brzozowski (pbrzozowski)]

We have checked servers for anomalies and we have not found anything interesting.

[Oleksiy Molchanov (omolchanov)]

On the reverted env I can see nodes are online.

[Georgy Kibardin (gkibardin)]

When a deployment is aborted astute stops the process, reboots the nodes and waits until they are available via mcollective. However, by that time nailgun agent not yet reported node status. This problem is well known, however I would expect it fixed in a nearest future. By now the more reasonable solution is to add a step to the test which ensures nodes are online before continue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/419899

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419899
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=831399e9bd0bb72e28ddc02836406076a94a58bd
Submitter: Jenkins
Branch: stable/mitaka

commit 831399e9bd0bb72e28ddc02836406076a94a58bd
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Jan 13 11:27:29 2017 +0000

    Nodes should be online for redeploy

    If the deployment was stopped, we
    should wait until nodes states will be
    "online". After that we can start redeploy.

    Change-Id: Ic15087bed08f3773325fec5dc6a7e85968f89400
    Closes-Bug: 1643817

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421090

[Dmitry Belyaninov (dbelyaninov)]

9.x
passed #174

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/421206

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421206
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=829f46477ff8e72b2f1b5db6830836203d90cc5a
Submitter: Jenkins
Branch: master

commit 829f46477ff8e72b2f1b5db6830836203d90cc5a
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Jan 13 11:27:29 2017 +0000

    Nodes should be online for redeploy

    If the deployment was stopped, we
    should wait until nodes states will be
    "online". After that we can start redeploy.

    Change-Id: Ic15087bed08f3773325fec5dc6a7e85968f89400
    Closes-Bug: 1643817
    (cherry picked from commit 831399e9bd0bb72e28ddc02836406076a94a58bd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421090
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=0a4db0ffc66826d740b85add646f95579c35c5f9
Submitter: Jenkins
Branch: stable/newton

commit 0a4db0ffc66826d740b85add646f95579c35c5f9
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Jan 13 11:27:29 2017 +0000

    Nodes should be online for redeploy

    If the deployment was stopped, we
    should wait until nodes states will be
    "online". After that we can start redeploy.

    Change-Id: Ic15087bed08f3773325fec5dc6a7e85968f89400
    Closes-Bug: 1643817
    (cherry picked from commit 831399e9bd0bb72e28ddc02836406076a94a58bd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-qa 11.0.0.0rc1 release candidate.

[Ilya Bumarskov (ibumarskov)]

Verified on Fuel 10 build #1575
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.unlock_settings_tab_states/186/testReport/(root)/unlock_settings_tab_positive/