Bootstrap process is very slow in scale

[Sergii Golovatiuk]

Steps To Reproduce:
1. Create environment with 200 servers.
2. Bootstrap 200 nodes (using ipmi)
3. Wait

Expected Result:
All 200 nodes will bootstraped to Fuel master node as slaves nodes within 10 minutes without problems.

Observed Result:
Fuel required 2 hours to bootstrap 200 nodes. Some nodes timed out and required 3-4 reboot cycles (happened automatically) to finish bootstrap process. There are some performance issues in tftp protocol, and all servers can't bootstrap from one Fuel master node in parallel. Also tftp forked too many processes as every tftp client require single tftp process

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/249075
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1c2eac08d222e0ffef2a7bccb9402816179e17df
Submitter: Jenkins
Branch: master

commit 1c2eac08d222e0ffef2a7bccb9402816179e17df
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Nov 20 16:33:44 2015 +0100

    Use HTTP to download kernel and initital RAM disk

    As of version 3.70 (Fuel uses 4.02) pxelinux can download files
    via HTTP.

    DHCP option 210 sets prefix for all files that pxelinux downloads.

    Doc-Impact: ops-guide
    Closes-Bug: 1521759

    Change-Id: Idb7dfde333a0b533699ebb976d94d03b5ede295a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/258734

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/258734
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=03558ce6627a5adb417b1f6bbe1c0b3a853e8d6b
Submitter: Jenkins
Branch: master

commit 03558ce6627a5adb417b1f6bbe1c0b3a853e8d6b
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Nov 20 16:33:44 2015 +0100

    Use HTTP to download kernel and initital RAM disk

    As of version 3.70 (Fuel uses 4.05) pxelinux can download files
    via HTTP.

    DHCP option 210 sets prefix for all files that pxelinux downloads.

    Change-Id: I26b62414e1cc0193c75f954ed7ed76570a4bf2f6
    Doc-Impact: ops-guide
    Closes-Bug: 1521759

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260170

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/260170
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=10ed2f60c6352cd358e216adecbc695c1a255fa4
Submitter: Jenkins
Branch: master

commit 10ed2f60c6352cd358e216adecbc695c1a255fa4
Author: Maksim Malchuk <email address hidden>
Date: Mon Dec 21 21:43:18 2015 +0300

    Force DHCP option 210 only for iPXE capable hardware

    DHCP option 210 sets only for iPXE capable hardware, in other cases
    used default TFTP method.

    Change-Id: Id1f0d9e32487bcd902940311ed8d8bef4129fdaf
    Doc-Impact: ops-guide
    Closes-Bug: 1521759
    Partial-Bug: 1528186
    Partial-Bug: 1527474

[Ivan Lozgachev (ilozgachev)]

Cannot verify because our nodes boot with TFTP only. Moving to Fix Released.

[Nastya Urlapova (aurlapova)]

@Ivan, your comment about nodes is unclear, and other thing what did you meant under "our nodes" - some different a bunch of HW from scale lab?