Admin->Volumes page in Horizon times out

[Tomasz Adam Jaroszewski]

One of environment in facing problems with listing of volumes (all volumes) via horizon; section in admin -> volumes.
Using standard HAProxy approach - connection in terminated before it could be serve to customer (timeout) but using direct connection to apache request is being served.
Time it takes for such request to finish is 1307084ms.

Other sections of horizon works correctly, for example: page admin->instances is generated in few seconds.

Number of active nova instances: 620
Number of cinder volumes: 700
MOS: 4.0

From log files (dashboard) I can see that for each instance "GET" request is being made while list of all volumes are being generated (from horizon).
Time it takes for each such request it 3 seconds.

... dashboard-urllib3.connectionpool: INFO Starting new HTTP connection (1): 10.135....
... dashboard-urllib3.connectionpool: DEBUG "GET /v2/9ee5b56733c54275a6f3e4a27c775cdf/servers/6d993ea0-23c8-42d8-a111-2941b3484261 HTTP/1.1" 200 1865
... dashboard-openstack_dashboard.api.nova: DEBUG novaclient connection created using token "524b4ccd078547c7a753a1f1c28abb92" and url "http://10.135.XXX:8774/v2/9ee5b56733c54275a6f3e4a27c775cdf"
... dashboard-urllib3.connectionpool: INFO Starting new HTTP connection (1): 10.135....
... dashboard-urllib3.connectionpool: DEBUG "GET /v2/9ee5b56733c54275a6f3e4a27c775cdf/servers/a0f664bb-d369-48ab-a83b-438a33560d49 HTTP/1.1" 200 1865

# time nova list --all-tenants
real 0m10.501s
user 0m0.532s
sys 0m0.128s

# time cinder list --all-tenants
real 0m23.020s
user 0m0.396s
sys 0m0.088s

# dpkg -l |grep -i dashboard
ii murano-dashboard 0.4.fuel4.0~mira8 murano-dashboard
ii openstack-dashboard 1:2013.2.1.fuel4.0~mira5 django web interface to Openstack
ii openstack-dashboard-ubuntu-theme 1:2013.2.1.fuel4.0~mira5 Ubuntu theme for the Openstack dashboard

[Bogdan Dobrelya (bogdando)]

"Using standard HAProxy approach - connection in terminated before it could be serve to customer (timeout) but using direct connection to apache request is being served.
Time it takes for such request to finish is 1307084ms."

I guess half of an hour for request completion should not be considered as a SUCCESS. Shouldn't it be considered as a bug for Horizon?

[Alexander Dobdin (sanek-dobdin)]

The backport to 4.0 is needed

[Timur Sufiev (tsufiev-x)]

The primary suspect here is line https://github.com/openstack/horizon/blob/2015.1.0b3/openstack_dashboard/dashboards/project/volumes/tabs.py#L77 - here we suffer from same problem as in bug 1438325: the lack of ability to pass several instance_id to nova servers.list call.

[Timur Sufiev (tsufiev-x)]

Upstream bug is https://bugs.launchpad.net/nova/+bug/1442310

[Dmitry Mescheryakov (dmitrymex)]

The issue is not targeted to 6.1 because it requires deep redesign of Nova API. Timur started a discussion in upstream on how to proceed: http://lists.openstack.org/pipermail/openstack-dev/2015-April/061097.html

A quick hack was suggested in the discussion. We can implement that fix in 6.1, but:
  * we don't understand how much does it help. It might make no difference at all.
  * it will require significant time of scale lab to first reproduce the issue and then test how the fix works.

[Timur Sufiev (tsufiev-x)]

Moving to 8.0, because there is a good possibility that this issue is going to be solved by SearchLight which should be ready during Liberty-3. We'll try SearchLight for ourselves in 1-2 weeks.

[Paul Karikh (pkarikh)]

Related Nova spec is in progress: https://review.openstack.org/#/c/239286
Horizon upstream patch is still in progress, too: https://review.openstack.org/#/c/213260/

[Timur Sufiev (tsufiev-x)]

We've got these 2 patches merged:

https://review.fuel-infra.org/gitweb?p=openstack/horizon.git;a=commit;h=7ed7535a46007218d33f68bcbbfbf36cf02d6c47

https://review.fuel-infra.org/gitweb?p=openstack/horizon.git;a=commit;h=5019aab45e5be66cd7ae45cba73d4b6aa5acc5eb

Having them should have already reduced response times at Admin->Volumes. There is still problem with an inefficient call to Nova, see comment#4. Reassigning the bug to Nova team, so they will update their progress here.

[Timur Sufiev (tsufiev-x)]

For the above CRs we have MOS bug 1508713 and bug 1508427.

[Roman Podoliaka (rpodolyaka)]

No longer fixing Wishlist bugs in 8.0

[Roman Podoliaka (rpodolyaka)]

The spec did not make it to Mitaka, there is review for Newton - https://review.openstack.org/#/c/239286/ . Moving to 10.0.