Deployment was stuck as one node was stuck on reboot

[Sergii Golovatiuk]

On large deployment installation we had a situation when one node was stuck on reboot (20 minutes)

root@node-16:~# uptime -s
2015-03-31 16:58:09

though in astute.log I see

2015-03-31T18:27:32 debug: [535] 135c09a6-b082-40c9-9eaf-1da3d3af4e22: MC agent 'puppetd', method 'enable', results: {:sender=>"25", :statuscode=>0, :statusmsg=>"OK", :data=>{:output=>"Already enabled"}}
2015-03-31T18:27:33 debug: [535] Retry #1 to run mcollective agent on nodes: '16'

which means the reboot was issues somewhere around 16:25-26

We should add tolerate functions like what we do for provisioning.

[Łukasz Oleś (loles)]

Deployment fails if pre_deployment_action fails on any node. It doesn't fail during pre_deploy action and during deploy. I will prepare a fix

[Bogdan Dobrelya (bogdando)]

@Lukasz, do you have an update or fix on review? Could you please link, if any WIP?

[Bogdan Dobrelya (bogdando)]

Correct me please, if I'm wrong, but this issue should be fixed in the scope of the https://blueprints.launchpad.net/fuel/+spec/200-nodes-support, hence superseded and won't fix

[Łukasz Oleś (loles)]

It should, but we missed pre deploy actions. I'm working on it

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/183081

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183081
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=b09729c64b695b2e6fcc88c31843321759ec45d5
Submitter: Jenkins
Branch: master

commit b09729c64b695b2e6fcc88c31843321759ec45d5
Author: Łukasz Oleś <email address hidden>
Date: Wed May 13 03:19:16 2015 +0200

    Remove nodes which failed to provision

    Currently during provision some nodes may fail but provision
    will success. This failed nodes are causing pre deployment actions
    to fail.
    This change removes failed nodes from deployment info and from all tasks.
    It is safe to do because currently we allow only compute nodes to fail.

    Change-Id: I5c3b677ca49ad9d2fd93a6ca1f524edc91e0766d
    Closes-bug: #1438933