Keystone performance degradation in 9.1 MU

[Pavel]

Looks like we got Keystone's performance degradation in 9.1 MU.

Here you can see fresh test results for 9.0 (build 945)
---------------------------
<substitute this part with TestRail host address>/index.php?/runs/view/22023&group_by=cases:section_id&group_order=asc
---------------------------
Test run was performed on ENV-14 Lab of Performance QA Team.

And here you can see fresh test results for 9.1 RC2 (snapshot 315):
---------------------------
- short run: <substitute this part with TestRail host address>/index.php?/runs/view/22009&group_by=cases:section_id&group_order=asc
- long run: <substitute this part with TestRail host address>/index.php?/runs/view/22004&group_by=cases:section_id&group_order=asc
---------------------------
As you can see we have about 30% degradation for both authentication and validation time in compare to 9.0 release.
The results are almost the same for both runs. Test runs were performed on ENV-12 Lab of Performance QA Team.

The same tests were also performed nearly 2 weeks ago for the earlier 9.1 snapshot (form 14.09.2016-16:43:22)
---------------------------
<substitute this part with TestRail host address>/index.php?/runs/view/20843&group_by=cases:section_id&group_order=asc
---------------------------
No performance regression as you can see.
Test run was performed on ENV-13 Lab of Performance QA Team.

ENV-12, ENV-14 and ENV-13 have the same hardware and configuraton of controlers. Hardware description can be found here:
---------------------------
<substitute this part with JIRA host address>/wiki/display/QA/Scale+lab+HW
---------------------------

A bit later I'll provide results of test run for 9.1 on ENV-14 and comparative stats of hardware resources consuming for 9.0 and 9.1 during Keystone tests were run.

[Boris Bobrov (bbobrov)]

What is the ISO number in https://mirantis.testrail.com/index.php?/runs/view/20843&group_by=cases:section_id&group_order=asc&group_id=56995 where there was no performance regression?

[Pavel (p.petrov)]

http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-09-14-164322/
http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-09-14-172321/x86_64/

[Boris Bobrov (bbobrov)]

The only patch that could affect us is https://review.fuel-infra.org/#/c/26523/. Pavel, could you please run the performance tests with the patch reverted? I have prepared the revert: https://review.fuel-infra.org/#/c/26985/.

[Boris Bobrov (bbobrov)]

On the env provided by Pavel we have reverted both patches merged in the last 2 weeks, https://review.fuel-infra.org/26664 and https://review.fuel-infra.org/26523. It has not fixed the issue, which probably means that these 2 patches are not the root cause.

[Boris Bobrov (bbobrov)]

I have inserted some debug prints to keystone and we ran the tests. From the logs from keystone perspective 90-percentile is 0.255000s. Which is much lower than what Pavel's tool have.

Because of that, i will stop debugging keystone. Instead, we shall bisect 9.1 build to find the first bad build.

[Boris Bobrov (bbobrov)]

Patch is https://review.fuel-infra.org/#/c/27081/

[Max Yatsenko (myatsenko)]

There are patched that change number of threads of keystone to 1 :

https://review.openstack.org/#/c/381032/
https://review.openstack.org/#/c/381034/

[Boris Bobrov (bbobrov)]

All patches were merged

[Timur Nurlygayanov (tnurlygayanov)]

Performance team verified the fix on MOS 9.1 RC3.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/33560