user list is much slower in mitaka and newton

[Sam Morrison]

With Kilo doing a user-list on V2 or V3 would take approx. 2-4 seconds

In Mitaka it takes 19-22 seconds. This is a significant slow down.

We have ~9,000 users

We also changed from going under eventlet to moving to apache wsgi

We have ~10,000 project and this api (project-list) hasn't slowed down so I think this is something specific to the user-list api

[Henry Nash (henry-nash)]

Are these users stored in the keystone SQL backend, or in something like LDAP/AD?

[Sam Morrison (sorrison)]

Sorry that would've been helpful. We are using SQL (mariadb with galera to be precise)

[Steve Martinelli (stevemar)]

My gut is telling me shadow users may be the initial cause

[Sam Morrison (sorrison)]

I've traced this down to https://github.com/openstack/keystone/blob/stable/mitaka/keystone/identity/backends/sql.py#L230

the call to to_dict() slows things down significantly

[Sam Morrison (sorrison)]

I'm a little perplexed why it has to do with to_dict as this is also in Kilo. I thought it may have been something about also moving from mysqldb -> pymysql but it doesn't look like this affects things

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347552

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/347967

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/347968

[OpenStack Infra (hudson-openstack)]

Change abandoned by Boris Bobrov (<email address hidden>) on branch: master
Review: https://review.openstack.org/347552
Reason: not needed for master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/347975

[Ron De Rose (ronald-de-rose)]

This has already been fixed in master:
https://review.openstack.org/#/c/327817/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347975
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f8c65c10dfe55787e9d5755f4986e8808aa6aa28
Submitter: Jenkins
Branch: stable/mitaka

commit f8c65c10dfe55787e9d5755f4986e8808aa6aa28
Author: Ronald De Rose <email address hidden>
Date: Wed Jul 27 17:52:53 2016 +0000

    Change LocalUser sql model to eager loading

    This patch changes the LocalUser sql model to eager loading. Subquery
    loading is eager loading as the parents are loaded, using one additional
    SQL statement, which issues a JOIN to a subquery of the original
    statement, for each collection requested.

    Closes-Bug: 1606426
    Change-Id: I48965676ad6a796115caef5e90974cb617243223
    (cherry-picked from: 4b9384dea36755c0dc0256f7392bf8c1e13f6632)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347967
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=8246fc2c359e75ec436e53ae82c9a01ccf6831c0
Submitter: Jenkins
Branch: master

commit 8246fc2c359e75ec436e53ae82c9a01ccf6831c0
Author: Boris Bobrov <email address hidden>
Date: Wed Jul 27 00:07:25 2016 +0300

    Test number of queries on list_users

    Change-Id: I399b4a2309f387418844f91ade1a3e59e6a2129a
    Related-Bug: 1606426

[OpenStack Infra (hudson-openstack)]

Change abandoned by Boris Bobrov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/347968
Reason: not needed

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 9.2.0 release.