Bad default values for workers for some services

[Brent Eagles]

Note: a patch for this had been up for awhile but didn't have a bug reported for it. It's a performance issue for glance in tripleo.

The default value for Glance API workers is 0, resulting in a single process for dealing with API requests intended for testing and debugging and is unsuitable for production. A better default would be to leave the value unset so that a suitable default can be scaled according to system resources.

[Brent Eagles (beagles)]

Actually this might extend far beyond the glance api. Current hieradata is:

service_configs.yaml:glance::api::workers: 0
service_configs.yaml:glance::registry::workers: 0
service_configs.yaml:heat::api::workers: 0
service_configs.yaml:heat::api_cfn::workers: 0
service_configs.yaml:heat::api_cloudwatch::workers: 0
service_configs.yaml:heat::engine::num_engine_workers: 0
service_configs.yaml:keystone::wsgi::apache::workers: "%{::processorcount}"
service_configs.yaml:neutron::agents::metadata::metadata_workers: 0
service_configs.yaml:neutron::server::api_workers:
service_configs.yaml:neutron::server::rpc_workers:
service_configs.yaml:nova::api::metadata_workers: 0
service_configs.yaml:nova::api::osapi_compute_workers: 0
service_configs.yaml:nova::conductor::workers: 0

So unless the services where the worker count is 0 interpret it as something other than single process, this could suck.

On my 2 CPU controller node I have:

1 process for glance API
1 process for glance registry
1 process each for neutron agents (though I think that's what it should be)
1 process for neutron-ns-meta-data-proxy
1 process for nova scheduler
1 process for nova-consoleauth
1 process for nova-novncproxy
1 process for aodh listener, notifier and evaluator each.
1 process for cinder scheduler
1 process for ceilometer agent-scheduler, agent-notification, collect and polling each
1 process each for swift container-replicator, account-reaper, account-replicator, container-auditor, object-replicator, account-auditor, proxy-server, object-updater and container-updater
1 process for gnocchi-metricd

I'll check the puppet to see if any of these are processorcount and work my way backwards.

[Brent Eagles (beagles)]

I was curious as to why nova seemed to do the right thing. It might be a happy accident. The puppet module defaults workers to undef and the puppet code contains something like this:

  if $workers {
    nova_config {
      'conductor/workers': value => $workers;
    }
  }

Leaving it up to the conductor (or whatever) code to determine the default. I'm not sure which way is way is better: having processorcount as default in puppet or leaving it unset and letting the code decide.

[Brent Eagles (beagles)]

Patch in progress: https://review.openstack.org/#/c/350219/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350219
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e6ecdb8b181eac9a7295b33c9691afe24f00640c
Submitter: Jenkins
Branch: master

commit e6ecdb8b181eac9a7295b33c9691afe24f00640c
Author: Joe Talerico <email address hidden>
Date: Tue Aug 2 14:28:55 2016 -0400

    Glance worker count fix

    This patch changes the default value and type of the Glance worker
    configuration to allow it to be unset and allow a system dependent
    default to be used (e.g. processorcount or some derivative value). The
    previous default of 0 would result in a single self contained process,
    which while suitable for debugging and testing is not appropriate for
    production deployments.

    Partial-Bug: #1626126
    Change-Id: I58a6a72a581e7083e1dc4e5ca568fdd3fdd6cdf1

[Brent Eagles (beagles)]

While the glance fix has merged, I would like to finish going through the rest of the services that have worker count of 0 before marking this as "Fix Released".

[Brent Eagles (beagles)]

ccamacho is taking care of metricd here: https://review.openstack.org/#/c/374704/

[Carlos Camacho (ccamacho)]

IIRC if configured as 0 will be the processor count (This is inconsistent as it does not apply to all services :P ).
We shouldn't rely on this inconsistency and fix the value for all, maybe to processor count.

Would be cool to have them configurable in THT (Which will help us to reduce the footprint on CI)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/374915

[Brent Eagles (beagles)]

It seems that the neutron metadata agent was the only other service suffering from this at the moment.

Patch is here: https://review.openstack.org/#/c/374915/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374915
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=313b92012d7ce5ee09287dec2a4460a863c31940
Submitter: Jenkins
Branch: master

commit 313b92012d7ce5ee09287dec2a4460a863c31940
Author: Brent Eagles <email address hidden>
Date: Thu Sep 22 12:46:37 2016 -0230

    Neutron metadata agent worker count fix

    This patch changes the default value and type of the NeutronWorkers
    parameter, allowing it to be unset and let a system-dependent value to
    be used (e.g. processorcount or some derivate value).

    Change-Id: Ia385b3503fe405c4b981c451f131ac91e1af5602
    Closes-Bug: #1626126

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0rc2 release candidate.