Glance default workers is total overkill for modern servers

[Erno Kuvaja]

On modern servers we do see dozens of cores which is total overkill of Glance workers while each of them has eventlet pool size of 1000 threads. Maybe we should consider limiting the total number of workers unless sepcifically configured by deployer.

[Stuart McLaren (stuart-mclaren)]

All 1000 green threads in an eventlet pool will run on the same CPU. You only get true concurrency when you increase the number of workers.

For example if you use a single worker and have 1000 green threads performing SSL encryption/decryption (eg talking SSL to Swift) that single CPU will be a bottleneck.

It would be useful to be clearer about the pros/cons (ideally backed up with testing/data) of changing the default.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/543549

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543549
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=6f87c7d567c9006b139b01fd1ab3bb9420fb407d
Submitter: Zuul
Branch: master

commit 6f87c7d567c9006b139b01fd1ab3bb9420fb407d
Author: Erno Kuvaja <email address hidden>
Date: Mon Feb 12 16:58:03 2018 +0000

    Limit default workers to 8

    With modern servers it's not uncommon to see environments that has
    dozens of logical processors. Glance defaults workers to number of
    these logical processors seen by OS and each worker spins up pool
    of 1000 threads. Having dozens of these thread pools consuming
    resources is total overkill for most of the environments.

    This change limits the amount of workers used by default to 8 which
    should be plenty for most of the deployments. This limitation does
    not affect if more workers is specifically configured in the config
    file.

    Change-Id: I6fcf2f8a416e34c0c719e1abd73f0525e686be5e
    Closes-Bug: #1748916

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/544113

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/544113
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=ef841b6704bae4e82f6f8b3ec59d4c37565ddb76
Submitter: Zuul
Branch: stable/queens

commit ef841b6704bae4e82f6f8b3ec59d4c37565ddb76
Author: Erno Kuvaja <email address hidden>
Date: Mon Feb 12 16:58:03 2018 +0000

    Limit default workers to 8

    With modern servers it's not uncommon to see environments that has
    dozens of logical processors. Glance defaults workers to number of
    these logical processors seen by OS and each worker spins up pool
    of 1000 threads. Having dozens of these thread pools consuming
    resources is total overkill for most of the environments.

    This change limits the amount of workers used by default to 8 which
    should be plenty for most of the deployments. This limitation does
    not affect if more workers is specifically configured in the config
    file.

    Change-Id: I6fcf2f8a416e34c0c719e1abd73f0525e686be5e
    Closes-Bug: #1748916
    (cherry picked from commit 6f87c7d567c9006b139b01fd1ab3bb9420fb407d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 16.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 17.0.0.0b1 development milestone.