Number of ceilometer notification workers should not exceed 10

[Tee Ngo]

Description
-----------
The current setting of notification workers can result in an excessive number in system that has controller(s) with high number of cores. This should be limited to 10 max to avoid unnecessary system resources consumption and to improve vcpu scaling responsiveness. The degraded scaling performance due to excessive number of workers is caused by time taken to rebuild the transformer caches in all workers following a swact/service restart as well as the increased cpu sample age in these caches.

The setting should be revised and further reduced once ceilometer transformers have all been deprecated and transformation functionality moved to ceilometer backend.

Severity
--------
Major

Steps to reproduce
------------------
In a system with high-core (30+) controllers, execute vcpu scaling test case either repeatedly (e.g. 25 times) or right after a controller swact.

Expected behavior
-----------------
TC executed successfully

Actual behavior
---------------
Occasional timeout

Reproducibility
---------------
Infrequent

System configuration
--------------------
Multi-node with high core controllers

Timestamp/Logs
--------------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/599737

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/599737
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=86d60c3c298d5737acff61f3a104731c00ac7fb2
Submitter: Zuul
Branch: master

commit 86d60c3c298d5737acff61f3a104731c00ac7fb2
Author: Tee Ngo <email address hidden>
Date: Tue Sep 4 14:06:26 2018 -0400

    Limit number of notification workers to 10 max

    The current setting of ceilometer notification workers can result in
    an excessive number in multi-node system with high core controllers.
    It should be limited to 10 max to avoid unnecessary system resources
    consumption and to improve vcpu scaling responsiveness. The degraded
    vcpu scaling performance is caused by time taken to rebuild the
    transformer caches in all workers following a swact/service restart
    as well as the increased cpu sample age in these caches.

    Tests conducted: sanity, nightly sanity and 25 iterations of vcpu
    scaling tests in multi-node system with 32-core controllers.

    Closes-Bug: #1790679

    Change-Id: I47b7f14a75846fd9765e9a2ea2fb3ca3a6cad01a
    Signed-off-by: Tee Ngo <email address hidden>