network topology page doesn't scale

[Chenjun Shen]

We use openstack-ansbile project (branch 14.1.0, so basically newton) to deploy our openstack cloud.

How to reproduce problem:
In our case, we have a big project, which has 400+ servers, and 26 networks.
When we click project/network/network topology page, it times out.

In the code, we found a scalability problem. Please see these lines: https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/network_topology/views.py#L251-L259

When loading network topology page (in _get_servers function), it iterates servers variable, and for each running server(instance), will make a i_console.get_console() call, and it's only sequentially.

Imagine you have 1000 running servers in the project, each get_console() takes 200 milliseconds. So in the end in order to get all consoles, it will cost you 1000 x 200 (milliseconds) = 200 seconds. In openstack-ansible project, the default timeout for haproxy is only 50. So in the end it already times out for the user.

Something even worse, in the network topology diagram, it seems to have an auto refresh enabled. So every 20-30 seconds(probably), it fires all the requests to backend again.

I might have some suggestions.
Either we make this console link in network topology configurable,
or we fire the requests concurrently to backend?

Thanks,
Chenjun Shen

[Chenjun Shen (cshen)]

Or another suggested fix would be

if the number of server list > a fixed number (like: 20), the console link should not be requested and shown due to performance consideration.

[Akihiro Motoki (amotoki)]

The tag should be 'topology-view' instead of 'neutron'.

[Akihiro Motoki (amotoki)]

If the problem is limited to get_console, it would be nice to change the URL from the direct link (like /instances/<instance-url>/vnc) to a console to the console tab (like /instances/<id>/?tab=instance_details__console).

Another idea is to drop "Open Console". We can access the server console from the instance detail page, so it just adds one extra click and we lose just a bit.

[Akihiro Motoki (amotoki)]

The issue happens because we need to check what type console is available. The direct link varies depending on the available console types, so it forces us to call a number of API calls.

[Akihiro Motoki (amotoki)]

There seems several solutions on the console link.

* Drop "Open Console" link
* Replace "Open Console" link with the URL of the console tab of the detail page
* Fetch "Open Console" link dynamically when an instance detail tool tip is open

Also we can add some settings:
* Enable/Disable auto refresh
* (any other?)

[Akihiro Motoki (amotoki)]

It looks like a bit terrible, so marking this as High.

[Chenjun Shen (cshen)]

Hi Akihiro,

you have some good points.

Since console link can be found in instances page, it could be dropped as an option.

When the link can be dynamically generated, it will be better.

[Ying Zuo (yingzuo)]

We discussed the options at the weekly meeting today. Replacing the console link with a link to the console tab of the detail page seems to be favorable. However, I just realized that there's already a "view instance details" link to the details page, so we should just drop it instead of having two links going to the same panel. I will ask for user feedback during the upcoming summit.

[Chenjun Shen (cshen)]

Hi Ying Zuo,

I see, thanks for the information. In our private cloud, we've just dropped the console link, because it's redundant.

[Ying Zuo (yingzuo)]

Chenjun, I got the same user feedback at the Summit yesterday. We should go ahead and drop the link.

[Chenjun Shen (cshen)]

Hi Ying, all right, I will work on a patch to drop the link. Thanks for the info.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/518104

[Chenjun Shen (cshen)]

Hi Ying Zuo,

may you please have a look at https://review.openstack.org/518104 ?

Sam has some different opinions on this fix (drop the link). What could we do next? Make a new patch, or?

Thank you.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/524149

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/524149
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=974f0418efdf171132747ded4a556df7b7af03d1
Submitter: Zuul
Branch: master

commit 974f0418efdf171132747ded4a556df7b7af03d1
Author: Akihiro Motoki <email address hidden>
Date: Thu Nov 30 20:45:07 2017 +0900

    Redirect VM console dynamically

    Previously when a user accesses the network topology panel
    horizon resolves VM console URL when returning server data.
    This requires one API call to nova per VM, and this causes
    a scaling issue.

    Actually there is no need to resolves VM console URL when
    the network topology. A console URL corresponding to a requested
    VM can be resolved when a user actually accesses a console.
    This reduces the number of API calls drastically and addresses
    the scaling issue reported in the bug report.

    Change-Id: Icc667449e2988e6227012a6e899835a97ce0d738
    Closes-Bug: #1723142

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/529052

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/529055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/529052
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=4906c986b57eea9b53e5b0f6006917c72fb5fcc0
Submitter: Zuul
Branch: stable/pike

commit 4906c986b57eea9b53e5b0f6006917c72fb5fcc0
Author: Akihiro Motoki <email address hidden>
Date: Thu Nov 30 20:45:07 2017 +0900

    Redirect VM console dynamically

    Previously when a user accesses the network topology panel
    horizon resolves VM console URL when returning server data.
    This requires one API call to nova per VM, and this causes
    a scaling issue.

    Actually there is no need to resolves VM console URL when
    the network topology. A console URL corresponding to a requested
    VM can be resolved when a user actually accesses a console.
    This reduces the number of API calls drastically and addresses
    the scaling issue reported in the bug report.

    Change-Id: Icc667449e2988e6227012a6e899835a97ce0d738
    Closes-Bug: #1723142
    (cherry picked from commit 974f0418efdf171132747ded4a556df7b7af03d1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 12.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/529055
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=44f5760ced642a4fdc54d8e46e23f162323bda5e
Submitter: Zuul
Branch: stable/ocata

commit 44f5760ced642a4fdc54d8e46e23f162323bda5e
Author: Akihiro Motoki <email address hidden>
Date: Thu Nov 30 20:45:07 2017 +0900

    Redirect VM console dynamically

    Previously when a user accesses the network topology panel
    horizon resolves VM console URL when returning server data.
    This requires one API call to nova per VM, and this causes
    a scaling issue.

    Actually there is no need to resolves VM console URL when
    the network topology. A console URL corresponding to a requested
    VM can be resolved when a user actually accesses a console.
    This reduces the number of API calls drastically and addresses
    the scaling issue reported in the bug report.

    Change-Id: Icc667449e2988e6227012a6e899835a97ce0d738
    Closes-Bug: #1723142
    (cherry picked from commit 974f0418efdf171132747ded4a556df7b7af03d1)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Kolodyazhny (<email address hidden>) on branch: master
Review: https://review.openstack.org/518104
Reason: This review is > 4 months without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon ocata-eol release.