Sensible cap to worker-multiplier is needed

[Nobuto Murata]

worker-multiplier is a common option across multiple OpenStack charms. Most of the control plane related charms will be deployed into LXD containers because of higher density and better separations. However, nova-compute cannot be deployed into LXD of course by nature, so no cap will be applied to worker-multiplier if no value is set.

https://jaas.ai/nova-compute#charm-config-worker-multiplier
> worker-multiplier
> (float) The CPU core multiplier to use when configuring worker processes for this services e.g. metadata-api. By default, the number of workers for each daemon is set to twice the number of CPU cores a service unit has. When deployed in a LXD container, this default value will be capped to 4 workers unless this configuration option is set.

One example was that a customer had 150+ workers which ate almost all memory of the system and led to OOM killer. While users can set an explicit value through the charm option, sensible default cap is nice to have even for bare metal.

[Brett Milford (brettmilford)]

We have encountered this in a deployment using reserved hugepages.
350 1G pages were reserved, leaving 22G for hypervisor processes.
Around 14G of this was consumed by metadata-api processes which in combination with the other system processes was leaving little free memory and resulting in memory fragmentation.
The outcome was that qemu was unable to allocate order 6 pages when launching VMs (which were to be backed by hugepages).
I expect in setups using reserved hugepages the effects of this bug would be more pronounced and occur more often.

[Andrea Ieri (aieri)]

subscribed field-high given the impact this bug has on environments using hugepages

[Celia Wang (ziyiwang)]

Current value we used in customers' cloud is worker-multiplier="0.1". (Brett and me tried to use 4 but that's not too helpful)
So a more sensible default value is needed.

[Nobuto Murata (nobuto)]

https://github.com/juju/charm-helpers/pull/553

[James Page (james-page)]

PR looks good and has been updated - will merge today.

I've dropped field-high and added field-medium for this bug as there is a documented workaround using the existing configuration option but this is still a good feature to get landed for our next release.

[Nobuto Murata (nobuto)]

For reactive charms (update config.yaml to reflect the change):
https://review.opendev.org/c/openstack/charm-layer-openstack-api/+/778839

For a classic charm, nova-compute as the example:
https://review.opendev.org/c/openstack/charm-nova-compute/+/778838
(I will copy the same to other classic charms once nova-compute one is approved)

[Aurelien Lourot (aurelien-lourot)]

@nobuto, thanks for doing this! Could you please also open a gerrit review to the charm-guide's release notes? Thanks!

[Nobuto Murata (nobuto)]

https://opendev.org/openstack/charm-nova-compute/commit/5db8e14d4d09726cace01e6cfe3fe40ce9c05913

[Nobuto Murata (nobuto)]

https://opendev.org/openstack/charm-layer-openstack-api/commit/34311a62e963d0ce903b7ddb9d70b8f071f71651

[Nobuto Murata (nobuto)]

https://review.opendev.org/q/topic:bug/1843011

[Billy Olsen (billy-olsen)]

charm-helpers fix was included in the v0.20.21 release.

~/work/charms/charm-helpers (master) $ git tag --contains 63dfd93
v0.20.21

[Nobuto Murata (nobuto)]

https://review.opendev.org/c/openstack/charm-guide/+/783781