Project Volumes page could be more optimised

[Julie Pichon]

With a large number of volumes, the project volume page can take a very long time to load.

It seems we call volumes_list() 3 times when loading the page.

1. Once to get the volumes list for populating the table - OK
    https://github.com/openstack/horizon/blob/e790ac070e/openstack_dashboard/dashboards/project/volumes/tabs.py#L65
2. Once when we retrieve the snapshots, to show the volumes they're associated with. The snapshots are hidden in a tab so we shouldn't need to get to them so early, this could easily be prevented by not preloading the tab
   https://github.com/openstack/horizon/blob/e790ac070e/openstack_dashboard/dashboards/project/volumes/tabs.py#L65
3. Another time when rendering the tabbed response, not quite sure why yet

[Julie Pichon (jpichon)]

This means that bug 1191006 (issues with table actions when using preload) will affect the Snapshots tab too.

[Lin Hua Cheng (lin-hua-cheng)]

I noticed this issue too while working on volume backups.

For now, I will probably try to implement the same way as snapshot (another call to volume list!) and tackle the performance separately rather than fixing it together with my patch.

Possible options:
We could memoize the call to _get_volumes(...), it fetches the same results anyway
or use the preload=False (if we get the bug mentioned above fixed)

[Julie Pichon (jpichon)]

The side-effects with preload=False are pretty unfortunate and visible to more people (as opposed to "only" people with lots and lots of volumes). Unfortunately I haven't had a chance to look into the preload bug yet, though definitely one we should get fixed :-)

Memoized could be a decent compromise though it didn't seem to work as I expected (still getting multiple calls).

[Julie Pichon (jpichon)]

Related: bug 1316793 (adding pagination to volumes) will probably have the biggest impact in terms of performance improvement for this page.

[Julie Pichon (jpichon)]

About #3 in the bug description: that last extra call to volume_lists comes from the quota check for the "Create volume", to see if the user is allowed to create a new volume.

https://github.com/openstack/horizon/blob/cc75a0f53c/openstack_dashboard/usage/quotas.py#L221
https://github.com/openstack/horizon/blob/cc75a0f53c/openstack_dashboard/dashboards/project/volumes/volumes/tables.py#L93

There's a couple of unfortunate things happening here. Since we're calling to the generic tenant_quota_usages() function we're also calling to Neutron, etc to get all the quotas. It's not as impactful as calling to the volume_list() multiple times though. It looks like we had to make our own quota calculations manually at the time but since then we've added a call to Cinder's limits APIs ( https://github.com/openstack/horizon/blob/cc75a0f53c/openstack_dashboard/api/cinder.py#L262 ) so we could at least move to using that for the allowed() method.

I suspect we're using tenant_quota_usages a bit too liberally in other places too (i.e. when we only need one quota rather than all of them).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104897

[Julie Pichon (jpichon)]

The performance issues should be tremendously mitigated by the fix for bug 1317606 in Cinder, which greatly reduces the length of time "cinder list" takes to answer. Handsomely the fix is already backported to Icehouse too.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104897
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=19ea565c2fe21695e1ed586ed3cf4bf626879337
Submitter: Jenkins
Branch: master

commit 19ea565c2fe21695e1ed586ed3cf4bf626879337
Author: Julie Pichon <email address hidden>
Date: Thu Jul 3 07:52:35 2014 +0100

    Remove use of "quota_tenant_usages" in Volumes page

    Replace it instead with a direct call to the Cinder limits. The
    generic usage quotas method performs some operations that can be
    costly when there is a large number of volumes.

    Change-Id: Ib5601d34a1854f4b190b3bc106504f384561fc30
    Partial-Bug: #1334608

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113913

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113913
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=8eaa1fb8ba29a197efc98cf2ec031af7284f7a32
Submitter: Jenkins
Branch: master

commit 8eaa1fb8ba29a197efc98cf2ec031af7284f7a32
Author: Julie Pichon <email address hidden>
Date: Wed Aug 13 15:35:10 2014 +0100

    Stop preloading the Volumes tabs

    It can take quite a long time to make all the API calls. No need to
    preload all the tabs before they're actually needed.

    Change-Id: Ia3b6b114f976429e82cfc5664c3172a9302703c9
    Closes-Bug: #1334608