20 nodes, not all are able to get provisioning info from cobbler

Bug #1355347 reported by Aleksandr Shaposhnikov on 2014-08-11
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Matthew Mosesohn
5.0.x
High
Meg McRoberts

Bug Description

When a lot of nodes trying to provisioning fuel not able to provide them with preseed info. At least it true for debian installation.

Steps to reproduce:

1. Add 20 nodes to the fuel (3 controllers + 17 computes). All of them also with ceph-osd's.
2. Start provisioning.

Observed behavior:
At least 2 nodes failed to retrieve preseed from fuel(cobbler).

Screenshot from one of the nodes in attachment.

This bug only affect generation of preseed for ubuntu. CentOS kickstart generation is fine (at least for 20 nodes)

Changed in fuel:
importance: Undecided → High
milestone: none → 5.1
Dima Shulyak (dshulyak) on 2014-08-12
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Andrew Woodward (xarses) on 2014-08-12
summary: - Node not able to get provisioning info from cobbler
+ 20 nodes, not all are able to get provisioning info from cobbler
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Vladimir Sharshov (vsharshov)
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/113867

Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Matthew Mosesohn (raytrac3r)
status: Confirmed → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/113942

Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Vladimir Kuklin (vkuklin)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Matthew Mosesohn (raytrac3r)
Changed in fuel:
importance: High → Medium
tags: added: release-notes
Changed in fuel:
importance: Medium → High

3 controllers + 7 computes deployment and 10 computes adds works fine so this workaround (10 and 10) works.

Checked fix with 20 nodes on Ubuntu and couldn't reproduce this bug anymore. Checked 5.0.1rc + fix for ubuntu preseed + fix for rsync.

Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Vladimir Sharshov (vsharshov)

Reviewed: https://review.openstack.org/113924
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=26b55bc341cb99bf9bfb9e1888693dded8663760
Submitter: Jenkins
Branch: master

commit 26b55bc341cb99bf9bfb9e1888693dded8663760
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 19:04:14 2014 +0400

    Add iops fact and add it to astute

    Astute needs to know how many nodes can
    be provisioned at a time, based on IOPS.

    Change-Id: If2cf35ac8195baf37cda50564b70ba3db836ccea
    Partial-Bug: #1355347

Reviewed: https://review.openstack.org/113920
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=50cde45c09d3c0228fb3d357f412d6024964ea15
Submitter: Jenkins
Branch: master

commit 50cde45c09d3c0228fb3d357f412d6024964ea15
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 19:01:24 2014 +0400

    Add sysstat to Fuel Master and astute

    sysstat provides iostat which is useful
    for calculating HDD throughput.

    Change-Id: I19bbee60b9ff509064843a468f47c0061736b370
    Partial-Bug: #1355347

Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Matthew Mosesohn (raytrac3r)

Reviewed: https://review.openstack.org/113867
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4ae17af4bf2cf775987de036b4fd09fb18b44767
Submitter: Jenkins
Branch: master

commit 4ae17af4bf2cf775987de036b4fd09fb18b44767
Author: Matthew Mosesohn <email address hidden>
Date: Wed Aug 13 16:07:07 2014 +0400

    Enable retry for Ubuntu installs

    This is a proactive solution to bug
    1355347, which allows retries for failed
    downloads of packages from apt mirror
    during installation.

    Change-Id: I848a86ef27f9ad8235024996b7417a30689daa4f
    Partial-Bug: #1355347

Reviewed: https://review.openstack.org/113942
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=35c4403067c550cfbd556afcdde251d732d7ed4f
Submitter: Jenkins
Branch: master

commit 35c4403067c550cfbd556afcdde251d732d7ed4f
Author: Vladimir Sharshov <email address hidden>
Date: Wed Aug 13 19:46:20 2014 +0400

    Add splay sleep before reboot nodes

    Bug:
     Resolves concurrency issue when deploying
     large clusters. Uses formula of nodes /
     IOPS * 180 to determine how large the
     splay size should be.

    Refactoring:
    - speed up tests

    Co-Authored-By: Matthew Mosesohn <email address hidden>

    Depends on:
    - https://review.openstack.org/#/c/113924/
    - https://review.openstack.org/#/c/113920/

    Change-Id: Ieefc843ccba0cdb0441679c73d2e39d6729a4fb0
    Closes-Bug: #1355347

Changed in fuel:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers