reboot_plugin: Cluster failed on deploy because nodes are offline

[ElenaRossokhina]

Detailed bug description:
Found on CI https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.fuel_plugin_reboot/62/testReport/(root)/deploy_cluster_with_reboot_plugin/
Steps to reproduce:
    1. Revert snapshot with 5 nodes
    2. Download and install fuel-plugin-builder
    3. Create plugin with reboot task
    4. Build plugin and copy it in var directory
    5. Install plugin to fuel
    6. Create cluster and enable plugin
    7. Provision nodes
    8. Collect timestamps from nodes
    9. Deploy cluster (fail here)
    10. Check if timestamps are changed

or system test deploy_cluster_with_reboot_plugin

Expected results:
Deploy is successful
Actual result:
Deploy failed with following message 'Nodes "slave-02_compute_ceph-osd (id=1, mac=64:2f:76:17:82:c4),slave-01_controller_ceph-osd (id=2, mac=64:c1:99:2c:34:33),slave-03_compute (id=3, mac=64:46:17:fb:91:a7)" are offline. Remove them from environment and try again.'
It is possible, time limit for reboot was expired.

[ElenaRossokhina (esolomina)]

it's strange that slave-03_compute (id=3, mac=64:46:17:fb:91:a7) was offline
in test we suppose, that compute node is not rebooted by this task

[ElenaRossokhina (esolomina)]

New failed case https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.fuel_plugin_reboot/75/testReport/(root)/deploy_cluster_with_reboot_plugin/

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.fuel_plugin_reboot/82/testReport/(root)/deploy_cluster_with_reboot_plugin/deploy_cluster_with_reboot_plugin/

[Alexey Stupnikov (astupnikov)]

There was a bug #1617329 with the same symptoms: slave nodes just didn't finish reboot process in time. A time-out value was increased to fix this issue.

[Alexey Stupnikov (astupnikov)]

I have doublechecked everything and it looks like we have another situation at the moment (the reported problem was solved): tests are no longer failing because of offline nodes. Right now failures are caused by 500 error:

2016-11-24 22:29:09.533 ERROR [7f3540682880] (logger) Response code '500 Internal Server Error' for POST /api/clusters from 10.109.0.1:46530

This bug is invalid and a new one should be opened with new issue reported.

[Alexey Stupnikov (astupnikov)]

Opened new bug #1644794