20 nodes, not all are able to get provisioning info from cobbler

[Aleksandr Shaposhnikov]

When a lot of nodes trying to provisioning fuel not able to provide them with preseed info. At least it true for debian installation.

Steps to reproduce:

1. Add 20 nodes to the fuel (3 controllers + 17 computes). All of them also with ceph-osd's.
2. Start provisioning.

Observed behavior:
At least 2 nodes failed to retrieve preseed from fuel(cobbler).

Screenshot from one of the nodes in attachment.

[Aleksandr Shaposhnikov (alashai8)]

This bug only affect generation of preseed for ubuntu. CentOS kickstart generation is fine (at least for 20 nodes)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113867

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113920

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113924

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113942

[Aleksandr Shaposhnikov (alashai8)]

3 controllers + 7 computes deployment and 10 computes adds works fine so this workaround (10 and 10) works.

[Aleksandr Shaposhnikov (alashai8)]

Checked fix with 20 nodes on Ubuntu and couldn't reproduce this bug anymore. Checked 5.0.1rc + fix for ubuntu preseed + fix for rsync.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113924
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=26b55bc341cb99bf9bfb9e1888693dded8663760
Submitter: Jenkins
Branch: master

commit 26b55bc341cb99bf9bfb9e1888693dded8663760
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 19:04:14 2014 +0400

    Add iops fact and add it to astute

    Astute needs to know how many nodes can
    be provisioned at a time, based on IOPS.

    Change-Id: If2cf35ac8195baf37cda50564b70ba3db836ccea
    Partial-Bug: #1355347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113920
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=50cde45c09d3c0228fb3d357f412d6024964ea15
Submitter: Jenkins
Branch: master

commit 50cde45c09d3c0228fb3d357f412d6024964ea15
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 19:01:24 2014 +0400

    Add sysstat to Fuel Master and astute

    sysstat provides iostat which is useful
    for calculating HDD throughput.

    Change-Id: I19bbee60b9ff509064843a468f47c0061736b370
    Partial-Bug: #1355347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113867
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4ae17af4bf2cf775987de036b4fd09fb18b44767
Submitter: Jenkins
Branch: master

commit 4ae17af4bf2cf775987de036b4fd09fb18b44767
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 16:07:07 2014 +0400

    Enable retry for Ubuntu installs

    This is a proactive solution to bug
    1355347, which allows retries for failed
    downloads of packages from apt mirror
    during installation.

    Change-Id: I848a86ef27f9ad8235024996b7417a30689daa4f
    Partial-Bug: #1355347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113942
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=35c4403067c550cfbd556afcdde251d732d7ed4f
Submitter: Jenkins
Branch: master

commit 35c4403067c550cfbd556afcdde251d732d7ed4f
Author: Vladimir Sharshov <email address hidden>
Date: Wed Aug 13 19:46:20 2014 +0400

    Add splay sleep before reboot nodes

    Bug:
     Resolves concurrency issue when deploying
     large clusters. Uses formula of nodes /
     IOPS * 180 to determine how large the
     splay size should be.

    Refactoring:
    - speed up tests

    Co-Authored-By: Matthew Mosesohn <email address hidden>

    Depends on:
    - https://review.openstack.org/#/c/113924/
    - https://review.openstack.org/#/c/113920/

    Change-Id: Ieefc843ccba0cdb0441679c73d2e39d6729a4fb0
    Closes-Bug: #1355347