Performance issue on high concurrency

Bug #1157261 reported by Zhou Yuan
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
Critical
Aleksandr Shaposhnikov
6.0.x
Invalid
Critical
Aleksandr Shaposhnikov
6.1.x
Invalid
Critical
Aleksandr Shaposhnikov
OpenStack Identity (keystone)
Invalid
Wishlist
Unassigned

Bug Description

Im currently using Keystone as the auth-server in Swift.
We have done performance test on our solution and we found that performance of keystone+swift is quite low, only 200 ops/s. While with the same setup the throughput can be improved to 7k ops/s by replacing keystone with swauth
We found the keystone process is fully saturating one sandy bridge core. This looks like a scalability issue.
Would be good if keystone could scale up well on a multi-core server.

Tags: performance
Revision history for this message
Dolph Mathews (dolph) wrote :

100% agree; these benchmarks fall in line with my own.

I believe the biggest bottleneck is caused by our driver architecture and the division we maintain between them at the controller level (e.g., cross-driver calls are incredibly expensive for trivial lookups that could be done in SQL joins). I'm sure there are a lot of significant improvements we could make before addressing that issue, though (more specific & thus efficient driver calls; paging & filter hints at the driver level, etc).

Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
importance: Medium → Wishlist
Revision history for this message
Peter Feiner (pete5) wrote :

I am going to submit a review momentarily that enables multiprocessing in keystone (i.e., multiple keystone processes on a host).

Note that you can also improve your keystone performance by using the Memcached backend instead of the default database backend. See instructions here: http://pic.dhe.ibm.com/infocenter/tivihelp/v48r1/topic/com.ibm.sco.doc_2.2/t_memcached_keystone.html.

Also note that I wrote an article on OpenStack performance a while ago that touches on these Keystone performance issues: http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Performance

Changed in keystone:
assignee: nobody → Peter Feiner (pete5)
Revision history for this message
Dolph Mathews (dolph) wrote :

> I am going to submit a review momentarily

Just wanted to note that we have a little over a day left to propose this for Havana before https://wiki.openstack.org/wiki/FeatureProposalFreeze sets in for keystone

Revision history for this message
Peter Feiner (pete5) wrote :

Whoops! Forgot to paste the review link.

https://review.openstack.org/#/c/42967/

Revision history for this message
Dolph Mathews (dolph) wrote :

Not your fault! It looks like hudson failed to cross link & update status for some reason.

Changed in keystone:
status: Triaged → In Progress
Revision history for this message
Dolph Mathews (dolph) wrote :

Unassigning due to inactivity.

Changed in keystone:
assignee: Peter Feiner (pete5) → nobody
status: In Progress → Triaged
Changed in mos:
importance: Undecided → Critical
milestone: none → 6.0
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Alexander, if you want us to address performance issues in Keystone in MOS let us be more specific and formal. Please file a separate issue with specific steps to reproduce, so we can reliably say which performance scenarios we are going to address and what is expected to work as a result of fixing bug.

Changed in mos:
status: New → Incomplete
Changed in mos:
assignee: nobody → Aleksandr Shaposhnikov (alashai8)
summary: - performance issue on high concurrency
+ Performance issue on high concurrency
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

The bug is hanging in Incomplete for more than a month for MOS project. Hence moving to invalid.

David Stanek (dstanek)
tags: added: performance
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

This bug isn't very useful. Targeted bugs for specific performance increases would be better. Marking this as "Invalid" please open bugs that are targeted not "general performance is poor"

Changed in keystone:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.