get_xstatic_dirs do not process None values in BASE_XSTATIC_MODULES

[Ivan Udovichenko]

The issue was noticed during package build:
"""
./run_tests.sh --no-virtual-env --no-pep8 --verbosity=2

The run_tests script is deprecated and will be removed in the
Queens Release Cycle (13.0), in favor of tox.

Running Horizon application tests
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 303, in execute
    settings.INSTALLED_APPS
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/<<PKGBUILDDIR>>/horizon/test/settings.py", line 142, in <module>
    settings_utils.BASE_XSTATIC_MODULES, HORIZON_CONFIG
  File "/<<PKGBUILDDIR>>/openstack_dashboard/utils/settings.py", line 263, in get_xstatic_dirs
    for file in files:
TypeError: 'NoneType' object is not iterable
Running openstack_dashboard tests
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 303, in execute
    settings.INSTALLED_APPS
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/<<PKGBUILDDIR>>/openstack_dashboard/test/settings.py", line 19, in <module>
    from horizon.test.settings import * # noqa: F403,H303
  File "/<<PKGBUILDDIR>>/horizon/test/settings.py", line 142, in <module>
    settings_utils.BASE_XSTATIC_MODULES, HORIZON_CONFIG
  File "/<<PKGBUILDDIR>>/openstack_dashboard/utils/settings.py", line 263, in get_xstatic_dirs
    for file in files:
TypeError: 'NoneType' object is not iterable
Tests failed.
"""

This is a default configuration without additional changes.

http://paste.openstack.org/show/erxuvARQtXgWTmDnkECd/

Proposed fix:
https://review.openstack.org/517149

[Akihiro Motoki (amotoki)]

--no-virtual-env is passed to run_tests.sh. It seems python modules or related packages in your system causes this error.
If this bug does not happen when run_tests.sh is used with virtualenv, this is specific to your system and is not a horizon bug.
I tried run_tests.sh with virtualenv and confirmed it worked well, so I mark this as Incomplete.

[Ivan Udovichenko (iudovichenko)]

Ok, I made some investigation and found out that 2.1.0.1 version doesn't have MAIN defined in __init__.py -> https://github.com/openstack/xstatic-angular-gettext/blob/86c16c594194f5d29f75a347ba8ef70df83e1957/xstatic/pkg/angular_gettext/__init__.py
And I actually use this version 2.1.0.
But 2.3.8.0 already has it https://github.com/openstack/xstatic-angular-gettext/blob/2.3.8.0/xstatic/pkg/angular_gettext/__init__.py#L41

Another example xstatic-font-awesome package: https://github.com/openstack/horizon/blob/12.0.0/openstack_dashboard/utils/settings.py#L212
It has empty list and not 'None' value. And its MAIN not defined https://github.com/openstack/xstatic-font-awesome/blob/master/xstatic/pkg/font_awesome/__init__.py#L40

So I see no issues in setting empty list as anyway it is going to be redefined late: https://github.com/openstack/horizon/blob/12.0.0/openstack_dashboard/utils/settings.py#L252-L260

[Akihiro Motoki (amotoki)]

So, can we ignore a case where MAIN is not defined COMPLETELY SAFELY?

I am not a JavaScript guy and cannot say this is no problem or not.
Perhaps I am not a right person to decide this.

[Akihiro Motoki (amotoki)]

Pike horizon requires XStatic-Angular-Gettext>=2.3.8.0 [1]. 2.1.0.1 of Angular Gettext is not a requirement of pike horizon. We don't support older version of python packages. This is just due to mis-combination of python packages.

[1] https://github.com/openstack/horizon/blob/master/requirements.txt#L46

[Ivan Udovichenko (iudovichenko)]

I intentionally went through the list https://github.com/openstack/horizon/blob/stable/pike/openstack_dashboard/utils/settings.py#L184-L220
and as an example found xstatic-angular-bootstrap package with required 2.2.0.0 version https://github.com/openstack/horizon/blob/stable/pike/requirements.txt#L44 but which has MAIN defined https://github.com/openstack/xstatic-angular-bootstrap/blob/2.2.0.0/xstatic/pkg/angular_bootstrap/__init__.py#L41 and link to the PYPI https://pypi.python.org/packages/68/e9/37d291562aae8020fc0a959143f4089a716fc9726a27e19cc8611b9a0f20/XStatic-Angular-Bootstrap-2.2.0.0.tar.gz#md5=1d7d02580f022c36628f99fbf40e39c9
but the value for it is set -> https://github.com/openstack/horizon/blob/stable/pike/openstack_dashboard/utils/settings.py#L193 when we should have None there. As it was clearly stated: "If the xstatic module does define a MAIN then set the files list to None." -> https://github.com/openstack/horizon/blob/stable/pike/openstack_dashboard/utils/settings.py#L180-L181

But as you may see that 'files' variable still is going to be redefined: https://github.com/openstack/horizon/blob/stable/pike/openstack_dashboard/utils/settings.py#L252-L260

Another example is xstatic-font-awesome which doesn't have MAIN defined https://github.com/openstack/xstatic-font-awesome/blob/8a671c3f6f5cf0248c8c0cb6080db7f11c80bd41/xstatic/pkg/font_awesome/__init__.py but still has empty list https://github.com/openstack/horizon/blob/stable/pike/openstack_dashboard/utils/settings.py#L212 instead of None.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/517149
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=d16f750b36db3f220bca63e8fab509074e49ed6d
Submitter: Zuul
Branch: master

commit d16f750b36db3f220bca63e8fab509074e49ed6d
Author: Ivan Udovichenko <email address hidden>
Date: Thu Nov 2 04:22:34 2017 +0300

    Raise detailed exception

    This change allows to throw a detailed exception
    in case if 'files' list has 'None' value:
    """
    Exception occurred:
      File "/<<PKGBUILDDIR>>/openstack_dashboard/utils/\
        settings.py", line 263, in get_xstatic_dirs
        for file in files:
    TypeError: 'NoneType' object is not iterable
    """

    Closes-Bug: #1732567

    Change-Id: Iaf6282819c26c12ca24d19ef7dee00a5c8586418

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 13.0.0.0b2 development milestone.