Bootstrap process is very slow in scale

Bug #1521759 reported by Sergii Golovatiuk
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Sergii Golovatiuk

Bug Description

Steps To Reproduce:
1. Create environment with 200 servers.
2. Bootstrap 200 nodes (using ipmi)
3. Wait

Expected Result:
All 200 nodes will bootstraped to Fuel master node as slaves nodes within 10 minutes without problems.

Observed Result:
Fuel required 2 hours to bootstrap 200 nodes. Some nodes timed out and required 3-4 reboot cycles (happened automatically) to finish bootstrap process. There are some performance issues in tftp protocol, and all servers can't bootstrap from one Fuel master node in parallel. Also tftp forked too many processes as every tftp client require single tftp process

Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Sergii Golovatiuk (sgolovatiuk)
milestone: none → 8.0
tags: added: scale
description: updated
tags: added: area-library
Changed in fuel:
status: Confirmed → In Progress
tags: added: feature
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/249075
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1c2eac08d222e0ffef2a7bccb9402816179e17df
Submitter: Jenkins
Branch: master

commit 1c2eac08d222e0ffef2a7bccb9402816179e17df
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Nov 20 16:33:44 2015 +0100

    Use HTTP to download kernel and initital RAM disk

    As of version 3.70 (Fuel uses 4.02) pxelinux can download files
    via HTTP.

    DHCP option 210 sets prefix for all files that pxelinux downloads.

    Doc-Impact: ops-guide
    Closes-Bug: 1521759

    Change-Id: Idb7dfde333a0b533699ebb976d94d03b5ede295a

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/258734

Changed in fuel:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/258734
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=03558ce6627a5adb417b1f6bbe1c0b3a853e8d6b
Submitter: Jenkins
Branch: master

commit 03558ce6627a5adb417b1f6bbe1c0b3a853e8d6b
Author: Sergii Golovatiuk <email address hidden>
Date: Fri Nov 20 16:33:44 2015 +0100

    Use HTTP to download kernel and initital RAM disk

    As of version 3.70 (Fuel uses 4.05) pxelinux can download files
    via HTTP.

    DHCP option 210 sets prefix for all files that pxelinux downloads.

    Change-Id: I26b62414e1cc0193c75f954ed7ed76570a4bf2f6
    Doc-Impact: ops-guide
    Closes-Bug: 1521759

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/260170

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/260170
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=10ed2f60c6352cd358e216adecbc695c1a255fa4
Submitter: Jenkins
Branch: master

commit 10ed2f60c6352cd358e216adecbc695c1a255fa4
Author: Maksim Malchuk <email address hidden>
Date: Mon Dec 21 21:43:18 2015 +0300

    Force DHCP option 210 only for iPXE capable hardware

    DHCP option 210 sets only for iPXE capable hardware, in other cases
    used default TFTP method.

    Change-Id: Id1f0d9e32487bcd902940311ed8d8bef4129fdaf
    Doc-Impact: ops-guide
    Closes-Bug: 1521759
    Partial-Bug: 1528186
    Partial-Bug: 1527474

Revision history for this message
Ivan Lozgachev (ilozgachev) wrote :

Cannot verify because our nodes boot with TFTP only. Moving to Fix Released.

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Ivan, your comment about nodes is unclear, and other thing what did you meant under "our nodes" - some different a bunch of HW from scale lab?

Changed in fuel:
status: Fix Released → Fix Committed
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers