kolla

creating 100 vms via horizon on 64 node cluster returns error "max scheduling attempts reached 3"

Bug #1563664 reported by Manjeet Singh Bhatia
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Steven Dake
kolla newton-1 "n1"
Mitaka
Fix Released
Critical
Steven Dake
kolla mitaka-rc4 "mitaka-rc4"

Bug Description

when i create 100 vms (using horizon) on 3 node control 3 node network and 56 node compute get
nova-scheduler exception and image not found error aftrewards

trace of error message : http://paste.openstack.org/show/492389/

Revision history for this message
Hui Kang (huikang27) wrote :

Could you check which nova-compute node the scheduler selected to run the instance and post the message from that node? Maybe we can get more information. I am very interested in the scalability test. Thanks. - Hui

Revision history for this message
Steven Dake (sdake) wrote :

I think increasing the max scheduling retries to some larger number like 20 is the correct answer here in the jinja2 files.

Changed in kolla:
importance: Undecided → Critical
milestone: none → 1.1.0
status: New → Confirmed
milestone: 1.1.0 → 2.0.0
Steven Dake (sdake)
Changed in kolla:
assignee: nobody → Steven Dake (sdake)
Steven Dake (sdake)
Changed in kolla:
milestone: 2.0.0 → newton-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/300914

Changed in kolla:
status: Confirmed → In Progress
Steven Dake (sdake)
summary: - when i create 100 vms I get nova scheduler error
+ creating 100 vms via horizon on 64 node cluster returns error "max
+ scheduling attempts reached 3"
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/300914
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=8cdd8c4ec35e96d1f02d71fbf58a49f87961adc7
Submitter: Jenkins
Branch: master

commit 8cdd8c4ec35e96d1f02d71fbf58a49f87961adc7
Author: Steven Dake <email address hidden>
Date: Sun Apr 3 21:19:22 2016 -0400

    Increase max scheduling attempts to 10

    At high scale, such as 64 nodes with 13TB ram and 2600 cores, nova
    seems to struggle when scheduling 100+ VMs at the same time. The
    issue is unrelated to the database, as the error printed indicates
    the max_scheduling_attempts have been reached. Increase that value
    to something more fitting of a 100 node cluster.

    Change-Id: I8982d77c7c66db8f7c95b9fd73f58ceb66dbd723
    Closes-Bug: #1563664

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/303678

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/mitaka)

Reviewed: https://review.openstack.org/303678
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=28a3bb273af1721d85da972059c65b6242a0b5b4
Submitter: Jenkins
Branch: stable/mitaka

commit 28a3bb273af1721d85da972059c65b6242a0b5b4
Author: Steven Dake <email address hidden>
Date: Sun Apr 3 21:19:22 2016 -0400

    Increase max scheduling attempts to 10

    At high scale, such as 64 nodes with 13TB ram and 2600 cores, nova
    seems to struggle when scheduling 100+ VMs at the same time. The
    issue is unrelated to the database, as the error printed indicates
    the max_scheduling_attempts have been reached. Increase that value
    to something more fitting of a 100 node cluster.

    Change-Id: I8982d77c7c66db8f7c95b9fd73f58ceb66dbd723
    Closes-Bug: #1563664
    (cherry picked from commit 8cdd8c4ec35e96d1f02d71fbf58a49f87961adc7)

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/kolla 2.0.0.0rc4

This issue was fixed in the openstack/kolla 2.0.0.0rc4 release candidate.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 2.0.0

This issue was fixed in the openstack/kolla 2.0.0 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 1.1.0

This issue was fixed in the openstack/kolla 1.1.0 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 3.0.0.0b1

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.