Glance default workers is total overkill for modern servers

Bug #1748916 reported by Erno Kuvaja on 2018-02-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
High
Erno Kuvaja

Bug Description

On modern servers we do see dozens of cores which is total overkill of Glance workers while each of them has eventlet pool size of 1000 threads. Maybe we should consider limiting the total number of workers unless sepcifically configured by deployer.

Erno Kuvaja (jokke) on 2018-02-12
Changed in glance:
importance: Undecided → High
assignee: nobody → Erno Kuvaja (jokke)
Stuart McLaren (stuart-mclaren) wrote :

All 1000 green threads in an eventlet pool will run on the same CPU. You only get true concurrency when you increase the number of workers.

For example if you use a single worker and have 1000 green threads performing SSL encryption/decryption (eg talking SSL to Swift) that single CPU will be a bottleneck.

It would be useful to be clearer about the pros/cons (ideally backed up with testing/data) of changing the default.

Fix proposed to branch: master
Review: https://review.openstack.org/543549

Changed in glance:
status: New → In Progress

Reviewed: https://review.openstack.org/543549
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=6f87c7d567c9006b139b01fd1ab3bb9420fb407d
Submitter: Zuul
Branch: master

commit 6f87c7d567c9006b139b01fd1ab3bb9420fb407d
Author: Erno Kuvaja <email address hidden>
Date: Mon Feb 12 16:58:03 2018 +0000

    Limit default workers to 8

    With modern servers it's not uncommon to see environments that has
    dozens of logical processors. Glance defaults workers to number of
    these logical processors seen by OS and each worker spins up pool
    of 1000 threads. Having dozens of these thread pools consuming
    resources is total overkill for most of the environments.

    This change limits the amount of workers used by default to 8 which
    should be plenty for most of the deployments. This limitation does
    not affect if more workers is specifically configured in the config
    file.

    Change-Id: I6fcf2f8a416e34c0c719e1abd73f0525e686be5e
    Closes-Bug: #1748916

Changed in glance:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/544113
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=ef841b6704bae4e82f6f8b3ec59d4c37565ddb76
Submitter: Zuul
Branch: stable/queens

commit ef841b6704bae4e82f6f8b3ec59d4c37565ddb76
Author: Erno Kuvaja <email address hidden>
Date: Mon Feb 12 16:58:03 2018 +0000

    Limit default workers to 8

    With modern servers it's not uncommon to see environments that has
    dozens of logical processors. Glance defaults workers to number of
    these logical processors seen by OS and each worker spins up pool
    of 1000 threads. Having dozens of these thread pools consuming
    resources is total overkill for most of the environments.

    This change limits the amount of workers used by default to 8 which
    should be plenty for most of the deployments. This limitation does
    not affect if more workers is specifically configured in the config
    file.

    Change-Id: I6fcf2f8a416e34c0c719e1abd73f0525e686be5e
    Closes-Bug: #1748916
    (cherry picked from commit 6f87c7d567c9006b139b01fd1ab3bb9420fb407d)

tags: added: in-stable-queens

This issue was fixed in the openstack/glance 16.0.0.0rc2 release candidate.

Changed in glance:
milestone: none → queens-rc2

This issue was fixed in the openstack/glance 17.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers