[library] Performance issue in Keystone: let's use httpd / apache

[Mike Scherbakov]

Contact person: Boris Pavlovich
Issue: under constant load on Keystone, there is going to be an issue with eventlet over time - see results.html.gz below.

Gerrit change: https://review.openstack.org/#/c/90248/8, logs available:
http://logs.openstack.org/48/90248/8/check/check-rally-dsvm-rally/23c0d03/rally-plot/results.html.gz

Please follow http://docs.openstack.org/developer/keystone/apache-httpd.html docs on how to use Apache instead of eventlets.

[Boris Pavlovic (boris-42)]

In keystone logs multiply occurrences of eventlet tracebacks like this one http://paste.openstack.org/show/77411/

[Sergey Vasilenko (xenolog)]

I run the same rally/keystone task on the fresh deployed HA-env with one controller.
Deployment by FUEL gives better results, than devstack (see attach).

[Timur Nurlygayanov (tnurlygayanov)]

+1, need to use httpd/apache for Keystone to have an ability to scale Keystone API service.

[Sergey Vasilenko (xenolog)]

-1, we already use Ha-proxy as reverse proxy for keystone and another Openstack services. Any httpd server (Apache/Nginx/etc) do the same -- unlock working thread (by quick fetching response from keystone) and send it to the client as possible.

[Timur Nurlygayanov (tnurlygayanov)]

Sergey, yes, but how yu can increase the number of workers of Keystone API service if you want to support large workload?

In Nova, for exanple, we can set the number of API workers in the config.
For Keystone we have only one proccess, and this proccess allows only ~150 requests per seccond.
It is really bad 'peroformance' for production environments with 10 000 + users.

[Sergey Vasilenko (xenolog)]

In HA we have at least 3 controller nodes.

but I really missed that keystone hasn't "workers" option in his config.

[Timur Nurlygayanov (tnurlygayanov)]

Sergey, yes. And Apache/httpd will allow to configure number of workers for Keystone API on HA/nonHA environments.

[Oleksii Aleksieiev (alexzzman)]

Looks like this bug is duplicate for https://bugs.launchpad.net/fuel/+bug/1316857

[Matthew Mosesohn (raytrac3r)]

This is too major to change after Feature Freeze for 6.0. Moving to 'next' release.

[Vladimir Kuklin (vkuklin)]

It seems that Juno keystone already has multiple workers, so it's severity is lower for 6.0 release, but we might want to introduce frontend proxy for 6.1 release to support bigger environments

[Mike Scherbakov (mihgen)]

Ryan,
you've lowered priority of this issue. Do you have any info that it's not High for us in 6.1? Especially I'd love to know info from scale lab, so we decide based on data we have (not just feelings). Please don't lower priorities unless you provide reasoning (and please push others do the same). Thanks.

[Ryan Moe (rmoe)]

Mike,
It seems that as we can now have multiple workers (as mentioned by Vladimir) this bug isn't as severe. Also the related bug mentioned by Oleksii that prevents us from using Apache with Keystone at all only has a medium priority which is why I changed this one to match.

[Mike Scherbakov (mihgen)]

I've heard from services team, and from Keystone Core guy at the summit that it is still a problem, even if you have multiple workers, and the right solution is to put keystone behind Apache.
That's why I insist on real measurements, as I am still getting contradicting information.

[Ryan Moe (rmoe)]

That all makes sense. Moving back to high, sorry for the confusion.

[Dina Belova (dbelova)]

Folks, might this be related to the https://bugs.launchpad.net/mos/+bug/1413341 ?
We had issues on scale lab (Keystone stopped working after continuous high load on it) and it turned out, that the issue was in the incorrect order of eventlet monkey-patching of python logging.

Our issue seems to be gone after this, may you guys refresh the bug or have the additional investigation on it due to the things Keystone team has found?

[Bogdan Dobrelya (bogdando)]

Thank you for the feedback, @Dina. I set this bug to incomplete as it requires additional investigating

[Bogdan Dobrelya (bogdando)]

Confirmed status fits better, as we don't want an occasional expiration of this one.

[Bogdan Dobrelya (bogdando)]

Backports must be provided as well

[Vladimir Kuklin (vkuklin)]

Nope, backports will not be there as it requires modification of reference architecture. We will not configure apache for keystone in 6.0.x release.

[Bogdan Dobrelya (bogdando)]

Sorry for confusion, my point was that the *fix* should be backported in case of this issue is related with https://bugs.launchpad.net/mos/+bug/1413341

[Davanum Srinivas (DIMS) (dims-v)]

here's the note from keystone ptl to the operators list with lots of good reasons on why moving to Apache is a good idea and the deprecation path forward.

http://markmail.org/message/pur56ut2ikzpaglu

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/173314

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Alexander Didenko <email address hidden>
Review: https://review.fuel-infra.org/6251

[OSCI Robot (oscirobot)]

Fix proposed to branch: master
Review: https://review.fuel-infra.org/6251

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexander Didenko <email address hidden>
Review: https://review.fuel-infra.org/6326

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178554
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=ccecd5d22bcd44a46dfdfad31b2f0f259bd66cf9
Submitter: Jenkins
Branch: master

commit ccecd5d22bcd44a46dfdfad31b2f0f259bd66cf9
Author: Tatyana Leontovich <email address hidden>
Date: Wed Apr 29 11:34:39 2015 +0300

    Check if cookies has cfrtoken

    Try to get cfrtoken atribute from cookies,
    if we do not have such just try to login without it

    Related-Bug: 1313662
    Change-Id: I978a0cf2e4f96561113f0f05ad0d0947b45548e4

[Bogdan Dobrelya (bogdando)]

Folks, that is the status of this bug? The related commit looks stuck with -2 "Do not patch binaries inside puppet", any progress on this?

[Aleksandr Didenko (adidenko)]

We're blocked by this patch not being reviewed/merged:
https://review.fuel-infra.org/#/c/6326/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6326
Submitter: Dmitry Burmistrov <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: f8b2c05606e6476e58fb536dafcf3c32b3e27a43
Author: Alexander Didenko <email address hidden>
Date: Tue May 12 12:37:45 2015

Add keystone/wsgi.py to deb package

We need to ship wsgi keystone script and config example with the
package. We have them in RPM already, so need to change deb only.

Partial-bug: #1313662

Change-Id: If08509abd042461a521a1cda3298dcd71a9d28c5

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexander Didenko <email address hidden> on branch: master
Review: https://review.fuel-infra.org/6251
Reason: Invalid

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/173314
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4b1a79095994169189d18e7f5849c2c089263fd0
Submitter: Jenkins
Branch: master

commit 4b1a79095994169189d18e7f5849c2c089263fd0
Author: Dmitry Ilyin <email address hidden>
Date: Fri Apr 3 20:39:08 2015 +0300

    Enable keystone wsgi support

    * New task to deploy Apache and configure MPM
    * Run keystone as Apache vhost
    * Pull wsgi threads and priority configuration from upstream
      change-id: Ib05ac81381e169845b44b2ef7cb810a4d5db17de
    * Pull custom file source for wsgi scripts from upstream
      change-id: I941bf8804982e9081812e076f7a736f413220047
    * Fix for keystone spec tests (use concat 1.2.1 in fixtures)
    * Use keystone.py wsgi script from packages instead of downloading
      it from puppet module. Requires deb package with this patch:
      https://review.fuel-infra.org/6251
      Until it's megred, we'll use upstream wsgi script for Debian

    Change-Id: I85008079b0e922a4518c696a097238500132fa04
    Closes-Bug: 1313662

[Alexander Nevenchannyy (anevenchannyy)]

Verified on MOS 6.1 ISO #429
Steps to Verify:
root@node-1:~# ps -ef | grep keystone
keystone 17914 17911 0 14:47 ? 00:00:01 /usr/sbin/apache2 -k start
keystone 17915 17911 0 14:47 ? 00:00:01 /usr/sbin/apache2 -k start
root 30659 19123 0 15:06 pts/20 00:00:00 grep --color=auto keystone

[Bogdan Dobrelya (bogdando)]

Note, that this feature was reverted from the 6.1 as it doesn't fit into the HCF plans and its HA impact on memcache_pool backend was not tested as appropriate. The related bug reverted this is https://bugs.launchpad.net/fuel/+bug/1459977

[Boris Bobrov (bbobrov)]

Folks, any progress on this? I suggest to switch to apache as early as possible.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/204111

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/204111
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=67f1c37c54a2ba65829379eae505831e57c5776d
Submitter: Jenkins
Branch: master

commit 67f1c37c54a2ba65829379eae505831e57c5776d
Author: Aleksandr Didenko <email address hidden>
Date: Tue Jul 21 17:24:49 2015 +0300

    Run keystone under Apache wsgi

    Configure keystone under Apache wsgi.

    Change-Id: Iad9324cf1646965d67b1c55c73c4650bdf42a5b5
    Closes-Bug: 1313662

[Igor (ipukha)]

Verified on MOS 7.0 ISO #294