Horizon taking a long time to compress static assets

[Mark Goddard]

Currently in Kolla Ansible CI we are hitting an issue where accessing the dashboard times out. We have a timeout of 100s. The container is up and running, with no errors. Checking the ps logs, we can see the CPU is 100% performing static asset compression:

root 31399 31333 31399 99.1 3.7 336828 302972 /var/lib/kolla/venv/bin/python /var/lib/kolla/venv/bin/manage.py compress --force

I did some binary-chopping of Python packages between the Stein and Train centos-source-horizon images. The culprit seems to be PyScss 1.3.7. In the train image it is 1.3.4. On my laptop, this ~doubles the time required for static asset compression from 35s to 1m10s.

[Mark Goddard (mgoddard)]

This affects Stein deploy jobs and Train upgrade jobs (due to Stein images).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/717782

[Radosław Piliszek (yoctozepto)]

Affects kolla - flaky images.

[Mark Goddard (mgoddard)]

I created a test patch that pins PyScss to 1.3.4 in the horizon image: https://review.opendev.org/717776. This reverted to a more sane compression time and confirms the source of the problem.

[Mark Goddard (mgoddard)]

Just to spell it out, the problem is PyScss 3.4.7.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717782
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=88c4f98f07fc9fc1a79597328ce9ecef731b44ac
Submitter: Zuul
Branch: stable/train

commit 88c4f98f07fc9fc1a79597328ce9ecef731b44ac
Author: Mark Goddard <email address hidden>
Date: Mon Apr 6 16:08:24 2020 +0100

    CI: Bump dashboard access timeout to 1000s

    The stein horizon image is currently taking a very long time (over 7
    minutes) to perform static asset compression.

    Change-Id: If059ffcd85a663687719dd7f88acb832e79f4dcc
    Related-Bug: #1871138

[Akihiro Motoki (amotoki)]

I tested pyScss 1.3.4 and 1.3.7 (with and without the C extension) on Ubuntu 18.04 , but I did not see any meaningful difference among all combination. All took around 16-18 secs. http://paste.openstack.org/show/791711/
Based on this result, I am marking this as Incomplete in horizon.

[Marcin Juszkiewicz (hrw)]

Difference between 1.3.7 and 1.3.4 on my Ryzen desktop (docker on nvme):

11:22 (82s) hrw@puchatek:~$ docker run -ti --rm kolla/centos-source-horizon:stein bash
()[root@6e7d787e9863 /]# . /var/lib/kolla/venv/bin/activate
(venv) ()[root@6e7d787e9863 /]# pip list|grep pyscss -i
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
django-pyscss 2.0.2
pyScss 1.3.7
(venv) ()[root@6e7d787e9863 /]# pip list|grep pyScss
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
pyScss 1.3.7
(venv) ()[root@6e7d787e9863 /]# time /usr/local/bin/kolla_extend_start >/dev/null

real 0m35.036s
user 0m21.941s
sys 0m12.760s
(venv) ()[root@6e7d787e9863 /]# exit

11:24 (105s) hrw@puchatek:~$ docker run -ti --rm kolla/centos-source-horizon:stein bash
()[root@f995e06476ec /]# . /var/lib/kolla/venv/bin/activate
(venv) ()[root@f995e06476ec /]# pip install pyscss==1.3.4
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in indexes: http://mirror.ord.rax.opendev.org:8080/pypi/simple
Collecting pyscss==1.3.4
  Downloading http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8f51c4f28b2b172366ccd0820b14bb947350df82428dfce381/pyScss-1.3.4.tar.gz (120 kB)
     |████████████████████████████████| 120 kB 101 kB/s
Requirement already satisfied: six in /var/lib/kolla/venv/lib/python2.7/site-packages (from pyscss==1.3.4) (1.12.0)
Requirement already satisfied: enum34 in /var/lib/kolla/venv/lib/python2.7/site-packages (from pyscss==1.3.4) (1.1.6)
Requirement already satisfied: pathlib in /var/lib/kolla/venv/lib/python2.7/site-packages (from pyscss==1.3.4) (1.0.1)
Building wheels for collected packages: pyscss
  Building wheel for pyscss (setup.py) ... done
  Created wheel for pyscss: filename=pyScss-1.3.4-cp27-cp27mu-linux_x86_64.whl size=161001 sha256=369669b7a7b4f4532ab3d6191914eb840bdb7a55949d4decfd7050d46b6f8e6f
  Stored in directory: /root/.cache/pip/wheels/13/ba/58/eaef1267b7333e6d9889ef1fe1c202729180311958124fbf42
Successfully built pyscss
Installing collected packages: pyscss
  Attempting uninstall: pyscss
    Found existing installation: pyScss 1.3.7
    Uninstalling pyScss-1.3.7:
      Successfully uninstalled pyScss-1.3.7
Successfully installed pyscss-1.3.4
(venv) ()[root@f995e06476ec /]# time /usr/local/bin/kolla_extend_...

[Marcin Juszkiewicz (hrw)]

pyScss 1.3.5:

real 0m38.207s
user 0m25.541s
sys 0m12.354s

So it looks like it was changed earlier.

[Akihiro Motoki (amotoki)]

I tried python 2.7 version in the same environment as for comment #7.
I used stable/stein for reference.

pyScss 1.3.7 with python 2.7

(py27) user@node4:~/src/opendev.org/openstack/horizon (stable/train)[20:47:01]$ time python manage.py compress
Compressing... done
Compressed 7 block(s) from 12 template(s) for 0 context(s).

real 0m32.828s
user 0m29.356s
sys 0m3.422s

pyScss 1.3.4 with python 2.7

(py27) user@node4:~/src/opendev.org/openstack/horizon (stable/train)[20:49:12]$ time python manage.py compress
Compressing... done
Compressed 7 block(s) from 12 template(s) for 0 context(s).

real 0m29.959s
user 0m26.658s
sys 0m3.245s

[Akihiro Motoki (amotoki)]

python 2.7 version took longer time than python 3 version, but the difference was just a couple of seconds.

What I can tell are:
- Using python3 improves the performance
- pyScss 1.3.7 (possible >=1.3.5) take more time but the difference is just a couple of seconds (2~4 secs)

I ran my tests on ubuntu 18.04 baremetal machines. If it depends on machine environments (container vs baremetal), I am not sure this is an issue tackled by the horizon team.

I am leaning to marking this as Invalid in horizon.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Dashboard (Horizon) because there has been no activity for 60 days.]