visit horizon failure because of import module failure

Bug #1361235 reported by Leslie Wang
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Fix Released
Medium
Richard Jones
OpenStack-Ansible
Fix Released
High
Donovan Francesco
osprofiler
Fix Released
Critical
Unassigned
python-mistralclient
Fix Released
Undecided
Dina Belova
tripleo
Fix Released
High
James Slagle

Bug Description

1. Use TripleO to deploy both undercloud, and overcloud, and enable horizon when building images.
2. Visit horizon portal always failure, and has below errors in horizon_error.log

[Wed Aug 20 01:45:58.441221 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] mod_wsgi (pid=5035): Exception occurred processing WSGI script '/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi'.
[Wed Aug 20 01:45:58.441273 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] Traceback (most recent call last):
[Wed Aug 20 01:45:58.441294 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
[Wed Aug 20 01:45:58.449979 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] self.load_middleware()
[Wed Aug 20 01:45:58.450000 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/django/core/handlers/base.py", line 44, in load_middleware
[Wed Aug 20 01:45:58.450556 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] for middleware_path in settings.MIDDLEWARE_CLASSES:
[Wed Aug 20 01:45:58.450576 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/django/conf/__init__.py", line 54, in __getattr__
[Wed Aug 20 01:45:58.454248 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] self._setup(name)
[Wed Aug 20 01:45:58.454269 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in _setup
[Wed Aug 20 01:45:58.454305 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] self._wrapped = Settings(settings_module)
[Wed Aug 20 01:45:58.454319 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/django/conf/__init
__.py", line 128, in __init__
[Wed Aug 20 01:45:58.454338 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] mod = importlib.import_module(self.SETTINGS_MODULE)
[Wed Aug 20 01:45:58.454350 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
[Wed Aug 20 01:45:58.462806 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] __import__(name)
[Wed Aug 20 01:45:58.462826 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_dashboard/settings.py", line 28, in <module>
[Wed Aug 20 01:45:58.467136 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] from openstack_dashboard import exceptions
[Wed Aug 20 01:45:58.467156 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_dashboard/exceptions.py", line 22, in <module>
[Wed Aug 20 01:45:58.467667 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] from keystoneclient import exceptions as keystoneclient
[Wed Aug 20 01:45:58.467685 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/__init__.py", line 28, in <module>
[Wed Aug 20 01:45:58.472968 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] from keystoneclient import client
[Wed Aug 20 01:45:58.472989 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/client.py", line 13, in <module>
[Wed Aug 20 01:45:58.473833 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] from keystoneclient import discover
[Wed Aug 20 01:45:58.473851 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/discover.py", line 19, in <module>
[Wed Aug 20 01:45:58.474163 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] from keystoneclient import session as client_session
[Wed Aug 20 01:45:58.474182 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/session.py", line 27, in <module>
[Wed Aug 20 01:45:58.482131 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] osprofiler_web = importutils.try_import("osprofiler.web")
[Wed Aug 20 01:45:58.482152 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/openstack/common/importutils.py", line 71, in try_import
[Wed Aug 20 01:45:58.482434 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] return import_module(import_str)
[Wed Aug 20 01:45:58.482452 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/openstack/common/importutils.py", line 57, in import_module
[Wed Aug 20 01:45:58.482475 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] __import__(import_str)
[Wed Aug 20 01:45:58.482495 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../osprofiler/__init__.py", line 23, in <module>
[Wed Aug 20 01:45:58.487936 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] utils.import_modules_from_package("osprofiler._notifiers")
[Wed Aug 20 01:45:58.487956 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] File "/opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../osprofiler/_utils.py", line 172, in import_modules_from_package
[Wed Aug 20 01:45:58.488588 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] __import__(module_name)
[Wed Aug 20 01:45:58.488615 2014] [:error] [pid 5035:tid 3038755648] [remote 10.74.104.27:54198] ValueError: Empty module name

3. After troubleshooting, seems like function import_modules_from_package at _utils.py of osprofiler has one bug when file's path is /opt/stack/venvs/openstack/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../osprofiler/_notifier/base.py.

In this case, below line will always create new_package = ... osprofiler, then cause the error.
      new_package = ".".join(root.split(os.sep)).split("....")[1]

One possible solution is to find the first "...." from right to left, then use that as module_name.

Revision history for this message
James Slagle (james-slagle) wrote :

i've done some looking into this as well. As correctly indentified, the issue is with the import_modules_from_package function in osprofiler._utils.

The main difference between how TripleO configures horizon and how devstack configures horizon is that in TripleO, horizon is installed in a python virtualenv along with it's dependencies. The following line in the django.wsgi file for Horizon adds a relative path to sys.path:

sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), '../..'))

osprofiler ends up getting imported from the venv using that relative path (with a "../..") in it, and thus triggering this bug.

Revision history for this message
James Slagle (james-slagle) wrote :

Here's a pretty simple reproducer:

[jslagle@teletran-1 ~]$ python
Python 2.7.5 (default, Jun 25 2014, 10:19:55)
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.path.insert(0, '/home/jslagle/code/github/slagle/openstack/osprofiler/osprofiler/../../osprofiler')
>>> import osprofiler
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jslagle/code/github/slagle/openstack/osprofiler/osprofiler/../../osprofiler/osprofiler/__init__.py", line 23, in <module>
    utils.import_modules_from_package("osprofiler._notifiers")
  File "/home/jslagle/code/github/slagle/openstack/osprofiler/osprofiler/../../osprofiler/osprofiler/_utils.py", line 172, in import_modules_from_package
    __import__(module_name)
ValueError: Empty module name

That value inserted into sys.path is a real path, and valid for entry into sys.path.

I'd work on a patch (in fact I wrote a failing test case), but I really don't understand the intent behind the split on "...." in this function in _utils.py.

Changed in tripleo:
importance: Undecided → High
status: New → Triaged
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (master)

Fix proposed to branch: master
Review: https://review.openstack.org/116721

Changed in tripleo:
assignee: nobody → James Slagle (james-slagle)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (master)

Reviewed: https://review.openstack.org/116721
Committed: https://git.openstack.org/cgit/openstack/tripleo-image-elements/commit/?id=41c9a1dfad23f8aee366afb6a0b20a6c57ec8f79
Submitter: Jenkins
Branch: master

commit 41c9a1dfad23f8aee366afb6a0b20a6c57ec8f79
Author: James Slagle <email address hidden>
Date: Mon Aug 25 16:41:23 2014 -0400

    Workaround broken Horizon

    Horizon is currently broken for all of TripleO. This is a temporary
    workaround until the bug is fixed.

    Change-Id: I68c0cf03bdd9ac615ddbb49d40675ae870a245f3
    Partial-Bug: 1361235

Changed in osprofiler:
importance: Undecided → Critical
status: New → In Progress
Changed in tripleo:
importance: Critical → High
Revision history for this message
James Polley (tchaypo) wrote :

The .... seems to have been introduced in https://review.openstack.org/#/c/50027/, but the notes on that change and the related bug aren't sufficient for me to be able to figure out what they're intended to do

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
Dina Belova (dbelova) wrote :

this bug also jumped recently as a result of osprofiler integration to the Mistral/its client. It's introduced to client requirements explicitly (not like in other project), so Horizon installations like this are affected since latest Mistral client release.

Changed in python-mistralclient:
assignee: nobody → Dina Belova (dbelova)
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/python-mistralclient 2.1.1

This issue was fixed in the openstack/python-mistralclient 2.1.1 release.

Changed in python-mistralclient:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/python-mistralclient 2.1.1 release.

Changed in horizon:
status: New → Triaged
importance: Undecided → Medium
milestone: none → ocata-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to horizon (master)

Fix proposed to branch: master
Review: https://review.openstack.org/397002

Changed in horizon:
assignee: nobody → Richard Jones (r1chardj0n3s)
status: Triaged → In Progress
Changed in openstack-ansible:
status: New → In Progress
assignee: nobody → Donovan Francesco (donovan-francesco)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_horizon (master)

Reviewed: https://review.openstack.org/396693
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_horizon/commit/?id=6e23b3b2f0b7c37112d6fb4ca61b3c89f5f809e5
Submitter: Jenkins
Branch: master

commit 6e23b3b2f0b7c37112d6fb4ca61b3c89f5f809e5
Author: Donovan Francesco (drifterza) <email address hidden>
Date: Fri Nov 11 18:40:36 2016 +0200

    Patch for osprofiler regression in django wsgi.

     - Normalise the path to resolve ../..
     - The "../.." fragment appears to be causing code in osprofiler/_utils.py
       some issues, so we'll resolve it for them.
     - https://review.openstack.org/#/c/397002/1

    Change-Id: I3b8270b92be5d6ec58a17aea1b08a0203a144370
    Partial-Bug: 1361235

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to horizon (master)

Reviewed: https://review.openstack.org/397002
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=f7b271d4cdd973b1e76cb0edbc91ba8f3425c9c2
Submitter: Jenkins
Branch: master

commit f7b271d4cdd973b1e76cb0edbc91ba8f3425c9c2
Author: Richard Jones <email address hidden>
Date: Mon Nov 14 16:29:28 2016 +1100

    Normalise the path to resolve ../..

    The "../.." fragment appears to be causing code in osprofiler/_utils.py
    some issues, so we'll resolve it for them.

    Change-Id: Ib443e7c64aae23246751df8769cdb1a2bd713e16
    Partial-Bug: 1361235

Changed in horizon:
status: In Progress → Fix Released
Revision history for this message
Dina Belova (dbelova) wrote :
Changed in osprofiler:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.