Provision has failed. Too many nodes failed to provision

[Dmitry Belyaninov]

After the https://bugs.launchpad.net/fuel/+bug/1644618 was fixed
few tests failed with message:

Task 'deploy' has incorrect status. error != ready, 'Provision has failed. Too many nodes failed to provision'

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.ha_tun_group_1/30

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.ha_tun_group_2/29

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.ha_tun_group_3/29

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.mixed_os_components/29

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.sahara_os_component/30

and other...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/415839

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415845

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Sharshov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/415839

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/415860

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/415861

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415845
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=b0752c7a78ca2acbe85330adde22fb1d37eb73eb
Submitter: Jenkins
Branch: master

commit b0752c7a78ca2acbe85330adde22fb1d37eb73eb
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Dec 30 12:45:13 2016 +0300

    Fix fail detection for provisioning

    Add test to prevent such behavior in future

    Change-Id: If833723f0301f2008e6aabfc888d0bdf693f4f2e
    Partial-Bug: #1653210

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/415862

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415862
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=782b2a30a6963c0f509891aa7eb2988399c92d51
Submitter: Jenkins
Branch: stable/mitaka

commit 782b2a30a6963c0f509891aa7eb2988399c92d51
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Dec 30 12:45:13 2016 +0300

    Fix fail detection for provisioning

    Add test to prevent such behavior in future

    Change-Id: If833723f0301f2008e6aabfc888d0bdf693f4f2e
    Partial-Bug: #1653210
    (cherry picked from commit b0752c7a78ca2acbe85330adde22fb1d37eb73eb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415861
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=e9fa6117b502f0a1cbf74a84f7dad6d177795395
Submitter: Jenkins
Branch: stable/newton

commit e9fa6117b502f0a1cbf74a84f7dad6d177795395
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Dec 30 12:45:13 2016 +0300

    Fix fail detection for provisioning

    Add test to prevent such behavior in future

    Change-Id: If833723f0301f2008e6aabfc888d0bdf693f4f2e
    Partial-Bug: #1653210
    (cherry picked from commit b0752c7a78ca2acbe85330adde22fb1d37eb73eb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415860
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=7c0485eb1a8c75a57936eaecce935091659429e9
Submitter: Jenkins
Branch: master

commit 7c0485eb1a8c75a57936eaecce935091659429e9
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Dec 30 14:11:36 2016 +0300

    Network problem tolerance puppet status check

    Connection between node and Astute can be lost some
    times, so we need more tries to get info about task
    status on node.

    Two changes:

    - instead of 1 try Astute will run 6 tries with 10
    timeout for every attempt;
    - it will process such behavior for puppet using separately
      retries: puppet_undefined_retries

    Instead of full puppet retry status retry is safety because
    it is idempotent.

    Puppet undefined retries can be setup using Astute config
    or sending undefined_retries in puppet task parameters same
    way as for usual retries. Most important thing: it will refresh
    to original value every time when Astute get defined answer.

    Change-Id: Ie86576a3400be5a6b11041c8e6acf89abf3bbd51
    Related-Bug: #1653210
    Closes-Bug: #1653737

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/417358

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/417359

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417359
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=1b2bf8eaeec647cb1810347ebd4bdf96b0055c8e
Submitter: Jenkins
Branch: stable/mitaka

commit 1b2bf8eaeec647cb1810347ebd4bdf96b0055c8e
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Dec 30 14:11:36 2016 +0300

    Network problem tolerance puppet status check

    Connection between node and Astute can be lost some
    times, so we need more tries to get info about task
    status on node.

    Two changes:

    - instead of 1 try Astute will run 6 tries with 10
    timeout for every attempt;
    - it will process such behavior for puppet using separately
      retries: puppet_undefined_retries

    Instead of full puppet retry status retry is safety because
    it is idempotent.

    Puppet undefined retries can be setup using Astute config
    or sending undefined_retries in puppet task parameters same
    way as for usual retries. Most important thing: it will refresh
    to original value every time when Astute get defined answer.

    Change-Id: Ie86576a3400be5a6b11041c8e6acf89abf3bbd51
    Related-Bug: #1653210
    Closes-Bug: #1653737
    (cherry picked from commit 7c0485eb1a8c75a57936eaecce935091659429e9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417358
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=0497a610d2deeee528ea3af68cd95b1b8165dd41
Submitter: Jenkins
Branch: stable/newton

commit 0497a610d2deeee528ea3af68cd95b1b8165dd41
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Dec 30 14:11:36 2016 +0300

    Network problem tolerance puppet status check

    Connection between node and Astute can be lost some
    times, so we need more tries to get info about task
    status on node.

    Two changes:

    - instead of 1 try Astute will run 6 tries with 10
    timeout for every attempt;
    - it will process such behavior for puppet using separately
      retries: puppet_undefined_retries

    Instead of full puppet retry status retry is safety because
    it is idempotent.

    Puppet undefined retries can be setup using Astute config
    or sending undefined_retries in puppet task parameters same
    way as for usual retries. Most important thing: it will refresh
    to original value every time when Astute get defined answer.

    Change-Id: Ie86576a3400be5a6b11041c8e6acf89abf3bbd51
    Related-Bug: #1653210
    Closes-Bug: #1653737
    (cherry picked from commit 7c0485eb1a8c75a57936eaecce935091659429e9)