Dashboard take 5-10 secs for OpenStack operations

[Junaid Ali]

Dashboard takes around 5-10secs for OpenStack operations (e.g opening instance's page etc) from project panel in admin tenant. Although, if we open any page from admin panel, it takes around 2-3 seconds. Similarly, if we open any page from any other tenant, 5-10 secs are consumed.
CLI commands seem to be working fine as they take around 1-3.5 secs.

Please let me know the logs to share. Neutron, Keystone, Nova logs doesn't show any errors regarding slowness in API calls.

OpenStack Release: Mitaka
Horizon version: 9.1.0

[Junaid Ali (junaidali)]

Charm configurations that I'm using are:

mysql:
    dataset-size: 4G
    max-connections: 20000
    root-password: "admin"
    sst-password: "admin"
    vip: "<MYSQL_VIP>"
    vip_cidr: <NETWORK_PREFIX>
    vip_iface: "eth0"
hacluster-mysql:
    corosync_transport: unicast
    cluster_count: 3
    monitor_host: "<MONITOR_HOST_IP>"
keystone:
    admin-password: "admin"
    admin-token: "admin"
    region: "serverstack"
    openstack-origin: "cloud:trusty-mitaka"
    vip: "<KEYSTONE_VIP>"
    worker-multiplier: "1"
hacluster-keystone:
    cluster_count: 3
    corosync_transport: unicast
    monitor_host: "<MONITOR_HOST_IP>"
openstack-dashboard:
    vip: "<DASHBOARD_VIP>"
    openstack-origin: "cloud:trusty-mitaka"
hacluster-dashboard:
    cluster_count: 3
    corosync_transport: unicast
    monitor_host: "<MONITOR_HOST_IP>"
nova-compute:
    enable-live-migration: "True"
    enable-resize: "True"
    openstack-origin: "cloud:trusty-mitaka"
    migration-auth-type: "ssh"
    manage-neutron-plugin-legacy-mode: False
nova-cloud-controller:
    network-manager: "Neutron"
    console-access-protocol: "novnc"
    openstack-origin: "cloud:trusty-mitaka"
    region: "serverstack"
    single-nova-consoleauth: "False"
    vip: "<NOVA_CLOUD_CONTROLLER_VIP>"
    worker-multiplier: "1"
hacluster-nova-cc:
    cluster_count: 3
    monitor_host: "<MONITOR_HOST_IP>"
    corosync_transport: unicast
neutron-api:
    openstack-origin: "cloud:trusty-mitaka"
    neutron-security-groups: "True"
    neutron-plugin: "plumgrid"
    manage-neutron-plugin-legacy-mode: False
    region: "serverstack"
    vip: "<NEUTRON_VIP>"
    worker-multiplier: "1"
hacluster-neutron-api:
    cluster_count: 3
    monitor_host: "<MONITOR_HOST_IP>"
    corosync_transport: unicast
glance:
    openstack-origin: "cloud:trusty-mitaka"
    region: "serverstack"
    vip: "<GLANCE_VIP>"
    worker-multiplier: "1"
hacluster-glance:
    cluster_count: 3
    monitor_host: "<MONITOR_HOST_IP"
    corosync_transport: unicast
rabbitmq-server:
    vip: "<RABBIT_MQ_VIP>"

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Richard Jones (r1chardj0n3s)]

The Horizon team is aware that those panels incur a significant delay when loading or performing some actions. We are actively migrating the interface over to a more client-side approach which reduces the per-action (page load or otherwise) overhead significantly. If you wish to see this in action, please look at the reimplemented Containers or Images panels.

The effort to complete those reimplementations is ongoing.

[Junaid Ali (junaidali)]

Sharing the logs that Narinder Gupta suggested to update with.

[Junaid Ali (junaidali)]

Output of ps -ef on controller node 1

[Junaid Ali (junaidali)]

Output of ps -ef on controller node 2

[Junaid Ali (junaidali)]

Output of ps -ef on controller node 3

[Junaid Ali (junaidali)]

screenshot of command 'top' on controller 1

[Junaid Ali (junaidali)]

screenshot of command 'top' on controller 2 and 3

[Junaid Ali (junaidali)]

output of ps -ef in each dashboard lxc

[Junaid Ali (junaidali)]

output of
$ juju get openstack-dashboard

[Akihiro Motoki (amotoki)]

I tested with my local env which has 10 instances in a single project with Queens development version. They are based on Chrome development tools.
The first component is the time required to load the main page. The second component is the time required to load the related things like (JS, header, image).

Project Overview : 1.02 sec + 1.46sec (= 2.48 sec)
Project Instances : 2.77sec + 1.58 sec (= 4.35 sec)
Admin Overview : 1.40 sec + 1.46 sec (= 2.86 sec)
Admin Instances : 2.08 sec + 1.29 sec (= 3.37 sec)

The result is not so bad compared to the reported result.

It seems horizon needs additional ~1.5 sec to load extra contents (like JS and so on).
The time required to process a main page depends on contents, but 10 sec looks longer.

[Akihiro Motoki (amotoki)]

Considering my test result, 10 sec sounds a bit longer so we would like to know more detail test conditions. My test environment is not so special as my machine is a couple of years old.
I think we need more condition to evaluate this bug. Let me mark this as Incomplete.

[Akihiro Motoki (amotoki)]

Due to the nature of Incomplete status, let's change the priority to Undecided (in horizon).