Ui Performance in Ocata when viewing Identity tabs is horrible

[Jay Jahns]

Hello,

The Identity dashboard, responsible for showing Domains, Users, Groups, Projects, and Roles, suffers terribly when viewing items like Projects.

Our private production runs about 435 projects, and slightly more than 350 users.

For example, to view all projects, the UI takes about 18-20 seconds to properly render. The API call to view projects: openstack project list, takes about 2 seconds.

It seems like rendering this data in Angular is the culprit, but I have not been able to pinpoint anywhere to indicate it.

We are using UWSGI and HTTP-Socket mode. I have a haproxy load balancer in front of this node, and I observed the same if not worse performance with using Nginx or Apache on the same node proxying the UWSGI.

This is my uwsgi load at idle:
https://gist.github.com/jjahns/79bc336c447a5718afff2edcf0f3bd83

This is my uwsgi load when the projects tab is being pulled:
https://gist.github.com/jjahns/abac52381e7eb77f86ab92ed55b86e28

The render time here was approximately 18 seconds. Only one CPU core was used, despite all of these processes, and it was pinned at 100% as well.

We've tried Apache with the same results. We've also tried using Nginx, and no yielded benefit.

If this is a configuration problem, then someone should explain how to correct.

[Jay Jahns (jjahns)]

I've added Keystone, in case this is keystone-related.

Basically, if the API is v2.0, then we support pagination on the Identity => Projects page, which can help with performance.

If the API is v3, we don't perform any pagination. Essentially, if someone has hundreds of projects, then this page is completely unusable, making it pointless to even use the Identity dashboards if > 100 records exist for any section.

There has to be better controls over pagination here. It is unnecessary load on the node running Horizon.

[Tracy Jones (tjones-i)]

Aside from any rendering slowness it does not appear to me that they keystone v3 api supports paging. So any admin with a large number of tenants is going to suffer.

[Jay Jahns (jjahns)]

Also I do not see a way to make the admin account view the project => overview by default, making the Identity section the most unusable part of the entire UI.

This needs attention. How are people currently managing their projects and users without having to resort to only the CLI?

[Ying Zuo (yingzuo)]

As you mentioned, pagination is not supported in keystone V3 at the moment. Thanks for trying to get the attention from keystone team on this. At the same time, you may want to turn on the filter_data_first setting for the projects panel in Horizon. https://docs.openstack.org/horizon/latest/configuration/settings.html#filter-data-first

[Jay Jahns (jjahns)]

Ying, filter_data_first is a stopgap at best. I've enabled it, but unless the domain admin knows exactly what to look for, they can't see the full list and browse.

You can log that as a workaround, but I'm really hesitant to suggest keeping that as a solution.

The question is, where does this get tackled? If keystone has no plans of adding pagination (I thought there was a blueprint on this floating around), then we need to mitigate this when the table is rendered. Some thought is required on that.

Either way, I guess folks with project lists > 500 are using the CLI for everything? While that is fine for me, if I intend to offload day-to-day to an operator, I'd rather them be able to go to a single pane of glass.

[Lance Bragstad (lbragstad)]

I have a couple configuration specific questions. Do you have keystone setup to cache anything [0]? Do you notice any sort of latency with token validation at all, or is the performance hit solely with listing projects? What token format

These could all be related to performance and we have ways to cope with them.~

First, you should consider using the Fernet token provider if you haven't already. Fernet tokens are non-persistent, thus they don't bloat the database. An idle deployment using UUID tokens will slow naturally an entire OpenStack deploym

If you are running with Fernet, caching is a huge help. Keystone specifically offers different levels of caching per subsystem. Since 99% of deployments have relatively static identity data, I recommend caching everything (even though we

In Ocata we improved performance for token validation [1]. If there were insane amounts of revocation events in the keystone database, you'd likely notice performance issues with any interaction that requires a valid token.

In Newton, we made a bunch of improvements around keystone caching implementation. We had to in order to safely rely on a non-persistent token format, since we're rebuilding so much context on every token validation request. That will li

[0] https://docs.openstack.org/keystone/latest/admin/identity-caching-layer.html
[1] https://review.openstack.org/#/c/382107/

[Lance Bragstad (lbragstad)]

I'm having a whole bunch of issues with LP today. I apologize for the formatting in comment #6. This was the intended message:

I have a couple configuration specific questions. Do you have keystone setup to cache anything [0]? Do you notice any sort of latency with token validation at all, or is the performance hit solely with listing projects? What token format
 are you using (fernet or uuid)? If you have the ability to inspect keystone database directly, can you share how large your keystone.revocation_event table is?

These could all be related to performance and we have ways to cope with them.~

First, you should consider using the Fernet token provider if you haven't already. Fernet tokens are non-persistent, thus they don't bloat the database. An idle deployment using UUID tokens will slow naturally an entire OpenStack deployment over time as keystone's token database grows.

If you are running with Fernet, caching is a huge help. Keystone specifically offers different levels of caching per subsystem. Since 99% of deployments have relatively static identity data, I recommend caching everything (even though we do take cache invalidation measures on state change).

In Ocata we improved performance for token validation [1]. If there were insane amounts of revocation events in the keystone database, you'd likely notice performance issues with any interaction that requires a valid token.

In Newton, we made a bunch of improvements around keystone caching implementation. We had to in order to safely rely on a non-persistent token format, since we're rebuilding so much context on every token validation request. That will likely help if you're not already using it.

[0] https://docs.openstack.org/keystone/latest/admin/identity-caching-layer.html
[1] https://review.openstack.org/#/c/382107/

[Lance Bragstad (lbragstad)]

This also seems similar to https://bugs.launchpad.net/keystone/+bug/1700852 which we landed a fix for last release. You're likely already using it if you have caching configured though.

[Lance Bragstad (lbragstad)]

I'm curious if there is any update on the performance suggestions from comment #7.

[Akihiro Motoki (amotoki)]

For horizon, does it make sense to implementation pseudo pagination in the horizon side? In the pseudo pagination, horizon APi wrapper retrieves all data from keystone v3 API and filter projects based on marker (i.e., ID). Of course it is not perfect it might be a reasonable workaround. (Note that if keystone returns projects in a random order it makes the thing worse.)