Performance issue related to S3 API or/and Keystone

[Alexander Petrov]

Detailed bug description:

Rally load tests related to S3 show the low performance probably closely connected to Keystone. It is necessary to find out the cause of that behavior. Here the link to a more detailed report.
https://docs.google.com/document/d/1I0djM15qHSX3gmIglJkWGL6vCcN8zjo86tE8q-PnjGM/edit

Description of the environment:

 MOS 8.0 with 6 nodes (CPU: 1 (1)HDD: 150.0 GB RAM: 3.0 GB)
 3 nodes - Controller
 3 nodes - Compute, Storage - Ceph OSD

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Petrov (apetrov-n)]

I have performed the simple test. Keystoneclient generates the concurrent queries from the one solely user (admin). It is only connection and no other operation. In Rally world this scenario looks like this:
 "KeystoneAPI.test_01": [
    {
      "runner": {
        "type": "constant",
        "concurrency": 600,
        "times": 4000
      }
]

Env:
3 controllers, 3 compute+storage
OpenStack Release: Mitaka on Ubuntu 14.04
Compute: QEMU
Network: Neutron with VLAN segmentation
Storage Backends:
   Ceph RBD for volumes (Cinder)
   Ceph RadosGW for objects (Swift API)
   Ceph RBD for ephemeral volumes (Nova)
   Ceph RBD for images (Glance)
Capacity: CPU (Cores): 3 (3) RAM 8.9 GB HDD 450.0 GB

The results of the test is shown here:
http://172.18.196.27/s3token/9.08.html#/KeystoneAPI.test_01/overview

In short words, I see that KS is not able to service (in that configuration) more ~500 connections.
I don’t know maybe it’s normal behavior for KS for such workload?
if some botlleneck exists then we should figure out where is the problem.

[Leontii Istomin (listomin)]

It seems it's the same issue as described here https://bugs.launchpad.net/mos/+bug/1583095
The root cause is rsyslog configuration - https://bugs.launchpad.net/fuel/+bug/1580200
Please make sure that you have this fix applied https://review.openstack.org/#/c/319458/

[Dina Belova (dbelova)]

Alexander, due to the comment https://bugs.launchpad.net/mos/+bug/1570797/comments/3 it looks like you need to try to apply the needed fix on 8.0 (the fix was developed for 9.0). Please contact Maintenance team if bug https://bugs.launchpad.net/fuel/+bug/1580200 can be fixed in 8.0 as well.

[Dina Belova (dbelova)]

One more moment - is it possible for you to reproduce the same issue on 9.0 ISO with https://review.openstack.org/#/c/319458/ fix included?

[Dina Belova (dbelova)]

Marking Invalid for 9.0 (at least for now) due to the fact, that there is a great confidence from Keystone team that https://review.openstack.org/#/c/319458/ should make this issue gone in MOS 9.0. Alex will recheck it additionally against 9.0 after June 14th. Also he'll communicate with maintenance team if this can be backported to 8.0 as well.

[Dina Belova (dbelova)]

Due to my previous comment, I'm marking this bug as confirmed (unless we'll prove it's not) for 8.0 and invalid due to the same fix in 10.0

[Rodion Tikunov (rtikunov)]

Code of 8.0 doesn't have a lines which should be deleted by patch [0].
As per [1] it is dangerous to backport other patches.

[0] https://review.openstack.org/#/c/319458/
[1] https://bugs.launchpad.net/fuel/+bug/1580200/comments/18