kolla-ansible

nova-scheduler workers is not managed

Bug #1873753 reported by Jeffrey Zhang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Mark Goddard
kolla-ansible 10.0.0 "ussuri"
Stein
Fix Released
Medium
Unassigned
kolla-ansible 8.1.1 "stein"
Train
Fix Released
Medium
Mark Goddard
kolla-ansible 9.1.0 "Train"
Ussuri
Fix Released
Medium
Mark Goddard
kolla-ansible 10.0.0 "ussuri"

Bug Description

nova scheduler process count is not managed

OpenStack Infra (hudson-openstack)
Changed in kolla-ansible:
assignee: nobody → Jeffrey Zhang (jeffrey4l)
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in kolla-ansible:
assignee: Jeffrey Zhang (jeffrey4l) → Mark Goddard (mgoddard)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/721198
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=8523ab569d007ca9fb01df351e8726e5c23c36c7
Submitter: Zuul
Branch: master

commit 8523ab569d007ca9fb01df351e8726e5c23c36c7
Author: Jeffrey Zhang <email address hidden>
Date: Mon Apr 20 16:13:08 2020 +0800

    Manage nova scheduler workers count

    Closes-Bug: 1873753
    Change-Id: Ibf75b3fd3090d47b50c71d17f6c12177d16442ab

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
John Garbutt (johngarbutt) wrote :

Just extra notes on this one, high nova-scheduler CPU is because of this bug.

Its complicated how it happens. There is a new task for nova-computes to request for nova-scheduler to update its instance cache, defaults to every 120 seconds (only required if you want affinity scheduler decisions to be "more efficient"). Now this message is sent to the nova-scheduler fanout queue, which then causes every worker to refresh its cache list. This quickly becomes quite a high load when you have 40 workers on every controller, and really not that many compute nodes.

Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → Medium
milestone: none → 10.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/723842

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/723843

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/723843
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=4b70bb52e60950800407ed6c0bfba87ff78cceb8
Submitter: Zuul
Branch: stable/stein

commit 4b70bb52e60950800407ed6c0bfba87ff78cceb8
Author: Jeffrey Zhang <email address hidden>
Date: Mon Apr 20 16:13:08 2020 +0800

    Manage nova scheduler workers count

    Closes-Bug: 1873753
    Change-Id: Ibf75b3fd3090d47b50c71d17f6c12177d16442ab
    (cherry picked from commit 8523ab569d007ca9fb01df351e8726e5c23c36c7)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/723842
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1c93d7b0303e9f700f2d89fb88eb8bdd0444d5c5
Submitter: Zuul
Branch: stable/train

commit 1c93d7b0303e9f700f2d89fb88eb8bdd0444d5c5
Author: Jeffrey Zhang <email address hidden>
Date: Mon Apr 20 16:13:08 2020 +0800

    Manage nova scheduler workers count

    Closes-Bug: 1873753
    Change-Id: Ibf75b3fd3090d47b50c71d17f6c12177d16442ab
    (cherry picked from commit 8523ab569d007ca9fb01df351e8726e5c23c36c7)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.