Mitaka: dashboard performance
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Environment: Openstack Mitaka on top of Leap 42.1, 1 control node, 2 compute nodes, 3-node-
Issue: Since switching to Mitaka, we're experiencing severe delays when accessing the dashboard - i.e. switching between "Compute - Overview" and "Compute - Instances" takes 15+ seconds, even after multiple invocations.
Steps to reproduce:
1. Install Openstack Mitaka, incl. dashboard & navigate through the dashboard.
Expected result:
Browsing through the dashboard with reasonable waiting times.
Actual result:
Refreshing the dashboard can take up to 30 secs, switching between views (e.g. volumes to instances) takes about 15 secs in average.
Additional information:
I've had a look at the requests, the Apache logs and our control node's stats and noticed that it's a single call that's taking all the time... I see no indications of any error, it seems that once WSGI is invoked, that call simply takes its time. Intermediate curl requests are logged, so I see it's doing its work. Looking at "vmstat" I can see that it's user space taking all the load (Apache / mod_wsgi drives its CPU to 100%, while other CPUs are idle - and no i/o wait, no system space etc.).
---cut here---
control1:/var/log # top
top - 10:51:35 up 8 days, 18:16, 2 users, load average: 2,17, 1,65, 1,48
Tasks: 383 total, 2 running, 381 sleeping, 0 stopped, 0 zombie
%Cpu0 : 31,7 us, 2,9 sy, 0,0 ni, 65,0 id, 0,3 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu1 : 13,1 us, 0,7 sy, 0,0 ni, 86,2 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu2 : 17,2 us, 0,7 sy, 0,0 ni, 81,2 id, 1,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu3 : 69,4 us, 12,6 sy, 0,0 ni, 17,9 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu4 : 14,6 us, 1,0 sy, 0,0 ni, 84,4 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu5 : 16,9 us, 0,7 sy, 0,0 ni, 81,7 id, 0,7 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu6 : 17,3 us, 1,3 sy, 0,0 ni, 81,0 id, 0,3 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu7 : 21,2 us, 1,3 sy, 0,0 ni, 77,5 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Mem: 65943260 total, 62907676 used, 3035584 free, 1708 buffers
KiB Swap: 2103292 total, 0 used, 2103292 free. 53438560 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6776 wwwrun 20 0 565212 184504 13352 S 100,3 0,280 0:07.83 httpd-prefork
1130 root 20 0 399456 35760 22508 S 5,980 0,054 818:13.17 X
1558 sddm 20 0 922744 130440 72148 S 5,316 0,198 966:03.82 sddm-greeter
20999 nova 20 0 285888 116292 5696 S 2,658 0,176 164:27.08 nova-conductor
21030 nova 20 0 758752 182644 16512 S 2,658 0,277 58:20.40 nova-api
18757 heat 20 0 273912 73740 4612 S 2,326 0,112 50:48.72 heat-engine
18759 heat 20 0 273912 73688 4612 S 2,326 0,112 4:27.54 heat-engine
20995 nova 20 0 286236 116644 5696 S 2,326 0,177 164:38.89 nova-conductor
21027 nova 20 0 756204 180752 16980 S 2,326 0,274 58:20.09 nova-api
21029 nova 20 0 756536 180644 16496 S 2,326 0,274 139:46.29 nova-api
21031 nova 20 0 756888 180920 16512 S 2,326 0,274 58:36.37 nova-api
24771 glance 20 0 2312152 139000 17360 S 2,326 0,211 24:47.83 glance-api
24772 glance 20 0 631672 111248 4848 S 2,326 0,169 22:59.77 glance-api
28424 cinder 20 0 720972 108536 4968 S 2,326 0,165 28:31.42 cinder-api
28758 neutron 20 0 317708 101812 4472 S 2,326 0,154 153:45.55 neutron-server
#######
control1:/var/log # vmstat 1
procs -------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 2253144 1708 53440472 0 0 4 60 4 4 11 1 88 0 0
0 0 0 2255588 1708 53440476 0 0 0 568 3063 7627 15 1 83 0 0
1 0 0 2247596 1708 53440476 0 0 0 144 3066 6803 14 2 83 0 0
1 0 0 2156008 1708 53440476 0 0 0 72 3474 7193 25 3 72 0 0
2 0 0 2131968 1708 53440484 0 0 0 652 3497 8565 28 2 70 0 0
3 1 0 2134000 1708 53440512 0 0 0 14340 3629 10644 25 2 71 2 0
2 0 0 2136956 1708 53440580 0 0 0 12 3483 10620 25 2 70 3 0
9 1 0 2138164 1708 53440596 0 0 0 248 3442 9980 27 1 72 0 0
4 0 0 2105160 1708 53440628 0 0 0 428 3617 22791 27 2 70 1 0
3 0 0 2093416 1708 53440644 0 0 0 1216 3502 7917 25 2 72 1 0
3 0 0 2096344 1708 53440636 0 0 0 72 3555 9216 25 2 73 0 0
6 0 0 2073564 1708 53440636 0 0 0 72 3587 11160 28 2 70 0 0
2 0 0 2070236 1708 53440636 0 0 0 432 3854 8160 26 4 70 0 0
1 0 0 2103628 1708 53440640 0 0 0 76 3407 7492 25 3 73 0 0
3 0 0 2100320 1708 53440636 0 0 0 1384 3383 7955 24 2 73 1 0
3 0 0 2100648 1708 53440644 0 0 0 216 3571 12276 33 2 65 0 0
1 0 0 2091808 1708 53440672 0 0 0 72 3734 12848 36 2 62 0 0
1 0 0 2092400 1708 53440700 0 0 0 0 4029 16556 23 1 77 0 0
1 0 0 2092400 1708 53440696 0 0 0 72 2938 6621 14 1 85 0 0
0 0 0 2092080 1708 53440696 0 0 0 704 3067 6958 15 1 83 1 0
0 0 0 2092048 1708 53440696 0 0 0 360 3222 7279 14 1 85 1 0
---cut here---
Current test, switching to "Volumes" took 7 secs loading http://
---cut here---
[01/Jun/
[01/Jun/
[01/Jun/
[01/Jun/
[01/Jun/
[01/Jun/
[01/Jun/
[01/Jun/
---cut here---
We did not see this with previous releases, we used Liberty before and upgraded to Mitaka.
Just for reference, we're running the following version of Horizon:
--- cut here ---
control1:
Name : openstack-dashboard
Version : 9.0.1~a0~dev12
Release : 2.1
Architecture: noarch
Install Date: Tue May 10 12:39:04 2016
Group : Development/
Size : 51400386
License : Apache-2.0
Signature : RSA/SHA256, Mon May 9 11:07:15 2016, Key ID 893a90dad85f9316
Source RPM : openstack-
Build Date : Mon May 9 11:06:31 2016
Build Host : build81
Relocations : (not relocatable)
Vendor : obs://build.
URL : http://
--- cut here ---
Changed in horizon: | |
status: | Expired → New |
Do you have any logs indicating how long the service requests etc are taking? Its going to be fairly difficult for us to narrow down the cause otherwise, if we are just told that a page load takes some time, as there is obviously a lot that occurs within that page.