| 2017-10-13 09:46:42 |
Chenjun Shen |
description |
We use openstack-ansbile project (branch 14.1.0, so basically newton) to deploy our openstack cloud.
How to reproduce problem:
In our case, we have a big project, which has 400+ servers, and 26 networks.
When we click project/network/network topology page, it times out.
In the code, we found a scalability problem. Please see these lines: https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/network_topology/views.py#L251-L259
When loading network topology page (in _get_servers function), it iterates servers variable, and for each running server(instance), will make a i_console.get_console() call, and it's only sequentially.
Imagine you have 1000 running servers in the project, each get_console() takes 200 milliseconds. So in the end in order to get all consoles, it will cost you 1000 x 200 (milliseconds) = 200 seconds. In openstack-ansible project, the default timeout for haproxy is only 50. So in the end it already times out for the user.
Something even worse, in the network topology diagram, it seems to have an auto refresh enabled. So every 20-30(probably), it fires all the requests to backend again.
I might have some suggestions.
Either we make this console link in network topology configurable,
or we fire the requests concurrently to backend?
Thanks,
Chenjun Shen |
We use openstack-ansbile project (branch 14.1.0, so basically newton) to deploy our openstack cloud.
How to reproduce problem:
In our case, we have a big project, which has 400+ servers, and 26 networks.
When we click project/network/network topology page, it times out.
In the code, we found a scalability problem. Please see these lines: https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/network_topology/views.py#L251-L259
When loading network topology page (in _get_servers function), it iterates servers variable, and for each running server(instance), will make a i_console.get_console() call, and it's only sequentially.
Imagine you have 1000 running servers in the project, each get_console() takes 200 milliseconds. So in the end in order to get all consoles, it will cost you 1000 x 200 (milliseconds) = 200 seconds. In openstack-ansible project, the default timeout for haproxy is only 50. So in the end it already times out for the user.
Something even worse, in the network topology diagram, it seems to have an auto refresh enabled. So every 20-30 seconds(probably), it fires all the requests to backend again.
I might have some suggestions.
Either we make this console link in network topology configurable,
or we fire the requests concurrently to backend?
Thanks,
Chenjun Shen |
|
