'Could not find kernel image' on 450 nodes [9.2]

[Sergey Galkin]

Steps to reproduce:
1. Install Fuel 9.0
2. Upgrade to 9.2 with repo http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2017-01-10-102420/x86_64
3. Discover 450 nodes and try to deploy cluster with 350 nodes

During provisioning 95 nodes gone offline.

Consoles offline servers in loop output:

Could not find kernel image: /images/ubuntu_1404_x86_64/linux
Could not find kernel image: /images/ubuntu_1404_x86_64/linux
Could not find kernel image: /images/ubuntu_1404_x86_64/linux

snapshot available in http://mos-scale-share.mirantis.com/fuel-snapshot-2017-01-13_08-55-52.tar.gz

[Vladimir Kozhukalov (kozhukalov)]

there are lines in astute.log

2017-01-12 14:34:07 INFO [14088] Starting OS provisioning for nodes: 776,777,778,779,780,781,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,800,801,802,803,804,805,806,808,809,810,813,814,816,817,818,819,820,821,822,823,8
24,825,826,828,831,832,833,834
...
2017-01-12 14:45:56 ERROR [14088] MCollective agents 'uploadfile' '788' didn't respond within the allotted time.

2017-01-12 14:45:56 ERROR [14088] fbe8c151-365e-4bc4-839c-12c4fd97b07b: file was not uploaded /tmp/provision.json on node 788: fbe8c151-365e-4bc4-839c-12c4fd97b07b: MCollective agents 'uploadfile' '788' didn't respond within the allotted time.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/420028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420028
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=570049ca1fde98ee09952075f72bc4f28b9d8b71
Submitter: Jenkins
Branch: stable/mitaka

commit 570049ca1fde98ee09952075f72bc4f28b9d8b71
Author: Vladimir Kozhukalov <email address hidden>
Date: Fri Jan 13 18:16:21 2017 +0300

    Move not provisioned nodes to error status

    When there are lot of nodes to provision and we provision
    them by chunks, we could fail in the middle due to "Too many
    nodes failed to provision". If so, we need to append those
    nodes where we did not started provision at all to the list
    of failed nodes. Otherwise, those nodes will be reported
    as 'provisioned' with progress = 100 and rebooted.
    But for some reasons we bind all nodes before starting provision
    to debian-installer profile in cobbler, and being rebooted
    these not provisioned nodes will fail to boot, because since
    7.0 we put empty files where cobbler expects debian-installer
    kernel and initrd files. :-)

    Change-Id: I2a401b80614ee7dd5a10931b9b50bcff066f790f
    Closes-Bug: #1656269

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/420654

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420660

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420654
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=64d62086e88460e29f5d9117a1f4c69d391d4bd0
Submitter: Jenkins
Branch: master

commit 64d62086e88460e29f5d9117a1f4c69d391d4bd0
Author: Vladimir Kozhukalov <email address hidden>
Date: Fri Jan 13 18:16:21 2017 +0300

    Move not provisioned nodes to error status

    When there are lot of nodes to provision and we provision
    them by chunks, we could fail in the middle due to "Too many
    nodes failed to provision". If so, we need to append those
    nodes where we did not started provision at all to the list
    of failed nodes. Otherwise, those nodes will be reported
    as 'provisioned' with progress = 100 and rebooted.
    But for some reasons we bind all nodes before starting provision
    to debian-installer profile in cobbler, and being rebooted
    these not provisioned nodes will fail to boot, because since
    7.0 we put empty files where cobbler expects debian-installer
    kernel and initrd files. :-)

    Change-Id: I2a401b80614ee7dd5a10931b9b50bcff066f790f
    Closes-Bug: #1656269
    (cherry picked from commit 570049ca1fde98ee09952075f72bc4f28b9d8b71)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420660
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=e05f66d12ef0fc9cdc71b5955a55978929a7cda8
Submitter: Jenkins
Branch: stable/newton

commit e05f66d12ef0fc9cdc71b5955a55978929a7cda8
Author: Vladimir Kozhukalov <email address hidden>
Date: Fri Jan 13 18:16:21 2017 +0300

    Move not provisioned nodes to error status

    When there are lot of nodes to provision and we provision
    them by chunks, we could fail in the middle due to "Too many
    nodes failed to provision". If so, we need to append those
    nodes where we did not started provision at all to the list
    of failed nodes. Otherwise, those nodes will be reported
    as 'provisioned' with progress = 100 and rebooted.
    But for some reasons we bind all nodes before starting provision
    to debian-installer profile in cobbler, and being rebooted
    these not provisioned nodes will fail to boot, because since
    7.0 we put empty files where cobbler expects debian-installer
    kernel and initrd files. :-)

    Change-Id: I2a401b80614ee7dd5a10931b9b50bcff066f790f
    Closes-Bug: #1656269
    (cherry picked from commit 570049ca1fde98ee09952075f72bc4f28b9d8b71)

[Michael Semenov (msemenov)]

9.2 is not certified with more than 200 nodes. So, set as Fix Released in 9.2.

[Leontii Istomin (listomin)]

The issue affects provisioning 50 nodes.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-astute 11.0.0.0rc1 release candidate.