Lower number of workers by default

[Peter Sabaini]

On containerized setups with multiple cpus and hyperthreading on the metals the containers can see a large number of cores. With the default worker multiplier settings this eg. results in 112 workers plus a leader processes on one of our installs. With HA we have then 3x113 processes serving in total, which is a bit of a waste (3x16G of RSS for one service, as an example)

While this can be tuned down I'd suggest to cap the number of workers when the worker-multiplier setting is on default to make this more sensible out of the box

[Ryan Beisner (1chb1n)]

+1 to the need for this. I think we should consider adding a cap as a config option.

This is an issue in testing and publishing bundles that are one-size-fits-all. For example, the openstack-base or openstack-on-lxd bundles may work great on 8c or 16c boxes, but require tuning down max workers for ~7 charm applications on machines with 48c so as not to OOM the hosts.

One alternative/additive view is that Juju should implement container resource limits. I agree that is needed for this and other reasons. However, that wouldn't give the max workers cap tunable that is desired here.

[Felipe Reyes (freyes)]

Another option could be allow this config option to be more than just a multiplier, so you could set a fixed number of threads for instance.

Possible values:

- FLOAT, behaves as it does today
- INTw , fixed number of workers
- INTw,FLOAT,INTw , this would be <minimum number of workers>,<multiplier for the cpus>,<maximum number of workers>, with this we'll have the chance to say "I want a minimum of 4 workers, but if there are 'lots' of core go ahead an used them without going over 12 workers". In the context of bundles and non homogeneous hardware would be helpful.

[James Page (james-page)]

I agree that capping the calculated value is probably a good idea; we recently did this for the dataset size configuration in percona-cluster (max out at 512M if the end-user does not provide any explicit configuration).

We did discuss this a week or so ago - I think by default we should use 2 x the CPU's or 4 workers, whichever is lowest; if the charm user sets worker-multiplier 0.5 or 8 (or anything else), then we will obey that blindly, but the default unset configuration will cap at 4 workers.

[Ryan Beisner (1chb1n)]

+1 to everything in comment #3.

[James Page (james-page)]

glance updates -

https://review.openstack.org/#/c/451732

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/460031

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/451732
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=074263eaa700f213adf5a0c1a306e96812107a92
Submitter: Jenkins
Branch: master

commit 074263eaa700f213adf5a0c1a306e96812107a92
Author: James Page <email address hidden>
Date: Thu Mar 30 10:58:11 2017 +0100

    Make worker-multiplier sane in container environments

    Resync charm-helpers to pickup the capped worker-multiplier
    changes when deploying in containers.

    Drop the default value for worker-multiplier of 2.0; this
    is now handled from within the codebase rather than via a
    default configuration value, reflecting the differing
    behaviours between container and non-container deployments.

    Change-Id: I418aac85c408b285cf6807ff4072e565898df399
    Closes-Bug: 1665270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460031
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=21a4e5beb1c833ce7380f239de25e03ef1936503
Submitter: Jenkins
Branch: master

commit 21a4e5beb1c833ce7380f239de25e03ef1936503
Author: James Page <email address hidden>
Date: Wed Apr 26 10:54:15 2017 +0100

    Cap workers in containers, fix admin/pubic skew

    Resync charm-helpers to pickup the latest code for calculation
    of worker process configuration, creating better default
    worker configuration when deploying in LXD containers.

    Switch the skew between public and admin processes to favour
    public 0.75/0.25 as the public API endpoints of a service will
    typically get a larger number of hits.

    Fixup unit test for minor behavioural change in charm-helpers.

    Change-Id: I4ab1d28f907ce29d5602b48ba7a438fc3690277c
    Closes-Bug: 1665270
    Closes-Bug: 1686049

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460442
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=083b0412d4ebe660c4663ecb433b2b1c31fd4b59
Submitter: Jenkins
Branch: master

commit 083b0412d4ebe660c4663ecb433b2b1c31fd4b59
Author: James Page <email address hidden>
Date: Thu Apr 27 09:31:12 2017 +0100

    Make worker-multiplier sane in container environments

    Resync charm-helpers to pickup the capped worker-multiplier
    changes when deploying in containers.

    Drop the default value for worker-multiplier of 2.0; this
    is now handled from within the codebase rather than via a
    default configuration value, reflecting the differing
    behaviours between container and non-container deployments.

    Change-Id: I528678680881d2612aafe4921f9e169839e31183
    Closes-Bug: 1665270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460441
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=b699e56699ac174a75af1bd257dfa35132231e3f
Submitter: Jenkins
Branch: master

commit b699e56699ac174a75af1bd257dfa35132231e3f
Author: James Page <email address hidden>
Date: Thu Apr 27 09:28:27 2017 +0100

    Make worker-multiplier sane in container environments

    Resync charm-helpers to pickup the capped worker-multiplier
    changes when deploying in containers.

    Drop the default value for worker-multiplier of 2.0; this
    is now handled from within the codebase rather than via a
    default configuration value, reflecting the differing
    behaviours between container and non-container deployments.

    Change-Id: Iec5fa21b0e1b377bcb22ad5193c84aa0ae525f16
    Closes-Bug: 1665270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460438
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=106686b9cdb4290f3de8cb3403e62679e1779f24
Submitter: Jenkins
Branch: master

commit 106686b9cdb4290f3de8cb3403e62679e1779f24
Author: James Page <email address hidden>
Date: Thu Apr 27 09:24:09 2017 +0100

    Make worker-multiplier sane in container environments

    Resync charm-helpers to pickup the capped worker-multiplier
    changes when deploying in containers.

    Drop the default value for worker-multiplier of 2.0; this
    is now handled from within the codebase rather than via a
    default configuration value, reflecting the differing
    behaviours between container and non-container deployments.

    Fixup amulet tests to use common helper for action
    execution.

    Change-Id: I7cac84fde5c43d1827f753e198b3f4a8e1e4151b
    Closes-Bug: 1665270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460443
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=d6165b16d55d5f11ce5c71a0e52ff43789a6c560
Submitter: Jenkins
Branch: master

commit d6165b16d55d5f11ce5c71a0e52ff43789a6c560
Author: James Page <email address hidden>
Date: Thu Apr 27 09:34:00 2017 +0100

    Make worker-multiplier sane in container environments

    Resync charm-helpers to pickup the capped worker-multiplier
    changes when deploying in containers.

    Drop the default value for worker-multiplier of 2.0; this
    is now handled from within the codebase rather than via a
    default configuration value, reflecting the differing
    behaviours between container and non-container deployments.

    Change-Id: Idb10dc73b357cc3ed774365bae92962b8dfd0bfb
    Closes-Bug: 1665270

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460439
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=b8cccb3372ad609a4f8581102f05f5b51f584e06
Submitter: Jenkins
Branch: master

commit b8cccb3372ad609a4f8581102f05f5b51f584e06
Author: James Page <email address hidden>
Date: Thu Apr 27 09:27:05 2017 +0100

    Make worker-multiplier sane in container environments

    Resync charm-helpers to pickup the capped worker-multiplier
    changes when deploying in containers.

    Drop the default value for worker-multiplier of 2.0; this
    is now handled from within the codebase rather than via a
    default configuration value, reflecting the differing
    behaviours between container and non-container deployments.

    Change-Id: Ia6ea43528c32c45bb3852e0a532aa6f758a4c80b
    Closes-Bug: 1665270