tune undercloud service workers similar to overcloud

[Joe Talerico]

Keystone on the Undercloud is not properly tuned

WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=1 threads=12 user=keystone

On the Overcloud it is

WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=12 threads=1 user=keystone

There might be other services to interrogate. I haven't totally diagnosed, but this could be why scaling out my cloud with heat convergence enabled failed.

[Luke Short (ekultails)]

Which way is preferred? Setting the processes or threads to the $cpu_count?

[Bogdan Dobrelya (bogdando)]

@Luke, IIRC the summary from the freezing heartbeats under WSGI topic, it should be always threads=1, like processes=12 threads=1

[Herve Beraud (herveberaud)]

I confirm Bogdan you remember correctly:
- https://bugs.launchpad.net/nova/+bug/1829062/comments/19
- https://bugzilla.redhat.com/show_bug.cgi?id=1711794
- https://github.com/openstack/puppet-nova/blob/stable/stein/manifests/wsgi/apache_api.pp#L55-L57
- https://bugs.launchpad.net/nova/+bug/1829062

[Luke Short (ekultails)]

Somewhat related, here's the original (and resolved) report of this problem affecting the Overcloud's performance: https://bugzilla.redhat.com/show_bug.cgi?id=1347305

This is no longer an issue for the Undercloud on newer versions of OpenStack but it may still affect Queens.

[Luke Short (ekultails)]

I can confirm that this is fixed for all services in the latest version of Queens.

[stack@undercloud ~]$ sudo grep -ir thread /etc/httpd/conf*
/etc/httpd/conf.d/10-keystone_wsgi_main.conf: WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone
...

[Radoslaw Smigielski (radoslaw-smigielski)]

I know this is an old bug and I know that the AMQP heartbeat could be lost when we have more than one thread what was reported and investigated here https://bugzilla.redhat.com/show_bug.cgi?id=1711794

But for services like:
- Keystone which doesn't really use AMQP
- Or all *_api type services
wouldn't it make sense to configure threads>1 ?

By doing it we could significantly reduce load on controllers or UC.