kolla

scale limits inappropriate for Kolla for database connections

Bug #1564275 reported by Steven Dake
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Steven Dake
kolla newton-1 "n1"
Mitaka
Fix Released
Critical
Steven Dake
kolla mitaka-rc3 "mitaka-rc3"

Bug Description

With a 64 node cluster of 13tb of ram and 2600 cores, it is only possible to start 75 virtual machines via horizon.

Manjeet measured the database connection usage and in an idle state this configuration uses 650 database connections. To create 1000 VMs, it took approximately 1400 database connections. To create 2500 virtual machines, 1600 database connections were in use.

Steven Dake (sdake)
Changed in kolla:
importance: Undecided → Critical
status: New → Confirmed
assignee: nobody → Steven Dake (sdake)
milestone: none → mitaka-rc3
milestone: mitaka-rc3 → newton-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/299791

Changed in kolla:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/299791
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=1fac190cec3c628c8a4c745659fcec090d952abd
Submitter: Jenkins
Branch: master

commit 1fac190cec3c628c8a4c745659fcec090d952abd
Author: Steven Dake <email address hidden>
Date: Thu Mar 31 00:40:01 2016 -0700

    Increase database connections to 10000

    Scale limit testing on a 64 node cluster with 13 TB RAM and 2600
    cores showed that atleast 1800 database connections are required
    to appropriately start 2000 virtual machines simulateounsly. Other
    documentation on the internet recommends larger values such as 8000,
    so we set to a larger value that should be able to handle a maxed out
    512GB ram per compute node cluster with all services enabled.

    Change-Id: I8767cf3fb04e066cc22e796c647e944b4e4a1742
    Closes-Bug: #1564275

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/300658

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/mitaka)

Reviewed: https://review.openstack.org/300658
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=65be6a2aee4aa019f59b2968f7378689ebdaa368
Submitter: Jenkins
Branch: stable/mitaka

commit 65be6a2aee4aa019f59b2968f7378689ebdaa368
Author: Steven Dake <email address hidden>
Date: Thu Mar 31 00:40:01 2016 -0700

    Increase database connections to 10000

    Scale limit testing on a 64 node cluster with 13 TB RAM and 2600
    cores showed that atleast 1800 database connections are required
    to appropriately start 2000 virtual machines simulateounsly. Other
    documentation on the internet recommends larger values such as 8000,
    so we set to a larger value that should be able to handle a maxed out
    512GB ram per compute node cluster with all services enabled.

    Change-Id: I8767cf3fb04e066cc22e796c647e944b4e4a1742
    Closes-Bug: #1564275
    (cherry picked from commit 1fac190cec3c628c8a4c745659fcec090d952abd)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 2.0.0

This issue was fixed in the openstack/kolla 2.0.0 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 1.1.0

This issue was fixed in the openstack/kolla 1.1.0 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 3.0.0.0b1

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.