Openstack Mitaka: can not access dashboard(internal server 500)

Bug #1573488 reported by Rock
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Expired
Undecided
Unassigned

Bug Description

Openstack Mitaka on Ubuntu 14.04.
It is a two-node environment installed following the guide:
http://docs.openstack.org/mitaka/install-guide-ubuntu/overview.html

When I tried to access http://controller/horizon, it hung there and below are the logs from /var/log/apache2/error.log.
Apache2 hangs when loading '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.

It looks like something related to dependency? Please take a look what else information I need to attach here.

Many thanks

#########################
[/var/log/apache2/error.log]
[Fri Apr 22 16:51:38.033769 2016] [authz_core:debug] [pid 30211:tid 140178471515904] mod_authz_core.c(802): [client 10.0.0.11:32905] AH01626: authorization result of Require all granted: granted
[Fri Apr 22 16:51:38.033870 2016] [authz_core:debug] [pid 30211:tid 140178471515904] mod_authz_core.c(802): [client 10.0.0.11:32905] AH01626: authorization result of <RequireAny>: granted
[Fri Apr 22 16:51:38.085433 2016] [:info] [pid 30208:tid 140178545293056] mod_wsgi (pid=30208): Create interpreter 'controller|/horizon'.
[Fri Apr 22 16:51:38.089348 2016] [core:trace1] [pid 30211:tid 140178471515904] util_script.c(601): [client 10.0.0.11:32905] Status line from script 'django.wsgi': 200 Continue
[Fri Apr 22 16:51:38.089471 2016] [:info] [pid 30208:tid 140178545293056] [remote 10.0.0.11:33762] mod_wsgi (pid=30208, process='horizon', application='controller|/horizon'): Loading WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Fri Apr 22 08:56:41.512825 2016] [:info] [pid 30208:tid 140178562078464] mod_wsgi (pid=30208): Daemon process deadlock timer expired, stopping process 'horizon'.
[Fri Apr 22 08:56:41.513111 2016] [:info] [pid 30208:tid 140178648561536] mod_wsgi (pid=30208): Shutdown requested 'horizon'.
[Fri Apr 22 08:56:46.514631 2016] [:info] [pid 30208:tid 140178291226368] mod_wsgi (pid=30208): Aborting process 'horizon'.
[Fri Apr 22 16:56:46.526999 2016] [core:error] [pid 30211:tid 140178471515904] [client 10.0.0.11:32905] End of script output before headers: django.wsgi
[Fri Apr 22 16:56:46.764577 2016] [:info] [pid 30205:tid 140178648561536] mod_wsgi (pid=30208): Process 'horizon' has died, restarting.
[Fri Apr 22 16:56:46.765623 2016] [:info] [pid 30380:tid 140178648561536] mod_wsgi (pid=30380): Starting process 'horizon' with uid=125, gid=133 and threads=10.
[Fri Apr 22 16:56:46.766640 2016] [:info] [pid 30380:tid 140178648561536] mod_wsgi (pid=30380): Initializing Python.
[Fri Apr 22 16:56:46.810913 2016] [:info] [pid 30380:tid 140178648561536] mod_wsgi (pid=30380): Attach interpreter ''.

####################
root@controller:~# cat /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi
import logging
import os
import sys
from django.core.wsgi import get_wsgi_application
from django.conf import settings

# Add this file path to sys.path in order to import settings
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), '../..'))
os.environ['DJANGO_SETTINGS_MODULE'] = 'openstack_dashboard.settings'
sys.stdout = sys.stderr

sys.path.append("/usr/share/openstack-dashboard/")

DEBUG = True

application = get_wsgi_application()

Revision history for this message
Itxaka Serrano (itxaka) wrote :

What is your apache config for horizon? Can you paste it to be sure that its ok?

Changed in horizon:
status: New → Incomplete
Revision history for this message
dico (bohli-dhekra) wrote :

HI did you resolve this problem !!!!! can you help i have a similar situation

Revision history for this message
Rock (zrockf) wrote :
Download full text (8.6 KiB)

Hi Itxaka,
Thanks for your reply.
I don't have specific apache config for horizon in /etc/apache2/sites-enabled/ as the installation guide suggests.
But here is the default setting:
root@controller:~# cat /etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80>
 # The ServerName directive sets the request scheme, hostname and port that
 # the server uses to identify itself. This is used when creating
 # redirection URLs. In the context of virtual hosts, the ServerName
 # specifies what hostname must appear in the request's Host: header to
 # match this virtual host. For the default virtual host (this file) this
 # value is not decisive as it is used as a last resort host regardless.
 # However, you must set it for any further virtual host explicitly.
 #ServerName www.example.com

 ServerAdmin webmaster@localhost
 DocumentRoot /var/www/html

 # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
 # error, crit, alert, emerg.
 # It is also possible to configure the loglevel for particular
 # modules, e.g.
 #LogLevel info ssl:warn

 ErrorLog ${APACHE_LOG_DIR}/error.log
 CustomLog ${APACHE_LOG_DIR}/access.log combined

 # For most configuration files from conf-available/, which are
 # enabled or disabled at a global level, it is possible to
 # include a line for only one particular virtual host. For example the
 # following line enables the CGI configuration for this host only
 # after it has been globally disabled with "a2disconf".
 #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

root@controller:~# cat /etc/apache2/apache2.conf
# This is the main Apache server configuration file. It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.4/ for detailed information about
# the directives and /usr/share/doc/apache2/README.Debian about Debian specific
# hints.
#
#
# Summary of how the Apache 2 configuration works in Debian:
# The Apache 2 web server configuration in Debian is quite different to
# upstream's suggested way to configure the web server. This is because Debian's
# default Apache2 installation attempts to make adding and removing modules,
# virtual hosts, and extra configuration directives as flexible as possible, in
# order to make automating the changes and administering the server as easy as
# possible.

# It is split into several files forming the configuration hierarchy outlined
# below, all located in the /etc/apache2/ directory:
#
# /etc/apache2/
# |-- apache2.conf
# | `-- ports.conf
# |-- mods-enabled
# | |-- *.load
# | `-- *.conf
# |-- conf-enabled
# | `-- *.conf
# `-- sites-enabled
# `-- *.conf
#
#
# * apache2.conf is the main configuration file (this file). It puts the pieces
# together by including all remaining configuration files when starting up the
# web server.
#
# * ports.conf is always included from the main configuration file. It is
# supposed to determine listening ports for incoming connections which can be
# customized anytime.
#
# * Configuration files in the mods-enabled/, conf-enabled/ and sites-enabled/
# directories contain particular configuration sn...

Read more...

Revision history for this message
Rock (zrockf) wrote :

Hi Dico,
I haven't solved the problem yet. I decided to back to Liberty release. It works fine.

Revision history for this message
Freerider (seanjohnmcgrath) wrote :

This seems to be a legitimate bug, if you try and follow the ubuntu mitaka install. Is this being looked at? Are there any pointers? I've run into the EXACT same issue. ubuntu 14.04.4 LTS.

Revision history for this message
Freerider (seanjohnmcgrath) wrote :

Hmm. Fixed. (at least in my environment).

root@mitakamaster:/etc/apache2/conf-enabled# cat openstack-dashboard.conf
WSGIScriptAlias /horizon /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi
WSGIDaemonProcess horizon user=horizon group=horizon processes=3 threads=10
WSGIProcessGroup horizon
WSGIApplicationGroup %{GLOBAL} <------ ADDED THIS
Alias /static /usr/share/openstack-dashboard/openstack_dashboard/static/
Alias /horizon/static /usr/share/openstack-dashboard/openstack_dashboard/static/
<Directory /usr/share/openstack-dashboard/openstack_dashboard/wsgi>
  Order allow,deny
  Allow from all
</Directory>

Restarted apache, and bingo. Dashboard access on ubuntu 14.04 / mitaka.

Revision history for this message
Aashish (aash-43) wrote :

Hi,
I'm facing the same issue after following mitaka openstack installation guide on Ubuntu 14.04.4 LTS,

Please find the apache2 logs below,
[Fri May 13 11:31:04.573431 2016] [:info] [pid 1066:tid 139768017549184] mod_wsgi (pid=1066): Attach interpreter ''.
[Fri May 13 11:31:12.477906 2016] [authz_core:debug] [pid 1083:tid 139767832377088] mod_authz_core.c(802): [client 192.168.1.140:57969] AH01626: authorization result of Require all granted: granted
[Fri May 13 11:31:12.477967 2016] [authz_core:debug] [pid 1083:tid 139767832377088] mod_authz_core.c(802): [client 192.168.1.140:57969] AH01626: authorization result of <RequireAny>: granted
[Fri May 13 11:31:12.478073 2016] [authz_core:debug] [pid 1083:tid 139767832377088] mod_authz_core.c(802): [client 192.168.1.140:57969] AH01626: authorization result of Require all granted: granted
[Fri May 13 11:31:12.478083 2016] [authz_core:debug] [pid 1083:tid 139767832377088] mod_authz_core.c(802): [client 192.168.1.140:57969] AH01626: authorization result of <RequireAny>: granted
[Fri May 13 11:31:12.492575 2016] [:info] [pid 1067:tid 139767914018560] mod_wsgi (pid=1067): Create interpreter 'controller|/horizon'.
[Fri May 13 11:31:12.494695 2016] [:info] [pid 1067:tid 139767914018560] [remote 192.168.1.140:1863] mod_wsgi (pid=1067, process='horizon', application='controller|/horizon'): Loading WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Fri May 13 06:06:14.575389 2016] [:info] [pid 1067:tid 139767930803968] mod_wsgi (pid=1067): Daemon process deadlock timer expired, stopping process 'horizon'.
[Fri May 13 06:06:14.575539 2016] [:info] [pid 1067:tid 139768017549184] mod_wsgi (pid=1067): Shutdown requested 'horizon'.
[Fri May 13 06:06:19.576133 2016] [:info] [pid 1067:tid 139767662200576] mod_wsgi (pid=1067): Aborting process 'horizon'.
[Fri May 13 11:36:19.584275 2016] [core:error] [pid 1083:tid 139767832377088] [client 192.168.1.140:57969] End of script output before headers: django.wsgi
[Fri May 13 11:36:19.813866 2016] [:info] [pid 1063:tid 139768017549184] mod_wsgi (pid=1067): Process 'horizon' has died, restarting.

Revision history for this message
Aashish (aash-43) wrote :

I tried solution suggested by Freerider (seanjohnmcgrath), it did not resolve the issue in my environment, I got a new exception with that change in /etc/apache2/conf-enabled# cat openstack-dashboard.conf file as shown below,

[Fri May 13 06:20:47.166292 2016] [:error] [pid 1777:tid 139671292704512] File "/usr/lib/python2.7/dist-packages/compressor/templatetags/compress.py", line 71, in render_offline
[Fri May 13 06:20:47.166393 2016] [:error] [pid 1777:tid 139671292704512] 'You may need to run "python manage.py compress".' % key)
[Fri May 13 06:20:47.166488 2016] [:error] [pid 1777:tid 139671292704512] OfflineGenerationError: You have offline compression enabled but key "337c10dadc650d2900f653d9efa97201" is missing from offline manifest. You may need to run "python manage.py compress".

Revision history for this message
xiaoqiyixin (379893594-s) wrote :

i have the same problem,and my error.log only have "End of script output before headers: django.wsgi"

my apache2 have no broblem,http://localhost is normal.

I tried solution suggested by #6 ,but it did not resolve the issue.

comandline service apache2 restart with follews:
root@controller:/etc/apache2/conf-enabled# service apache2 restart
 * Restarting web server apache2 [fail]
 * The apache2 configtest failed.
Output of config test was:
AH00526: Syntax error on line 4 of /etc/apache2/conf-enabled/openstack-dashboard.conf:
Invalid command 'WSGIApplicationGroup%{GLOBAL}', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.

error.log
[Sun May 15 07:49:37.296060 2016] [mpm_event:notice] [pid 10704:tid 139920269113216] AH00489: Apache/2.4.7 (Ubuntu) mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Sun May 15 07:49:37.296102 2016] [core:notice] [pid 10704:tid 139920269113216] AH00094: Command line: '/usr/sbin/apache2'
[Sun May 15 10:30:51.322057 2016] [mpm_event:notice] [pid 10704:tid 139920269113216] AH00491: caught SIGTERM, shutting down

Revision history for this message
Freerider (seanjohnmcgrath) wrote :

aaishish, Are you seeing this error? Trying running it?

[Fri May 13 06:20:47.166488 2016] [:error] [pid 1777:tid 139671292704512] OfflineGenerationError: You have offline compression enabled but key "337c10dadc650d2900f653d9efa97201" is missing from offline manifest. You may need to run "python manage.py compress".

xiaoqiyixin, post your entire openstack-dashboard.conf. You sure there is no typo/space?

Revision history for this message
Aashish (aash-43) wrote :

Freerider, Yes I see the below mentioned error when I tried the solution mentioned in #6,

[Thu May 19 04:26:58.202768 2016] [:error] [pid 19301:tid 140428522313472] OfflineGenerationError: You have offline compression enabled but key "337c10dadc650d2900f653d9efa97201" is missing from offline manifest. You may need to run "python manage.py compress".

When I try to access the dashboard from the browser, I'm getting below error,
OfflineGenerationError at /auth/login/
You have offline compression enabled but key "337c10dadc650d2900f653d9efa97201" is missing from offline manifest. You may need to run "python manage.py compress".
Request Method: GET
Request URL: http://192.168.1.167/horizon/auth/login/?next=/horizon/
Django Version: 1.8.7
Exception Type: OfflineGenerationError
Exception Value:
You have offline compression enabled but key "337c10dadc650d2900f653d9efa97201" is missing from offline manifest. You may need to run "python manage.py compress".
Exception Location: /usr/lib/python2.7/dist-packages/compressor/templatetags/compress.py in render_offline, line 71
Python Executable: /usr/bin/python
Python Version: 2.7.6
Python Path:
['/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../..',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client',
 '/usr/share/openstack-dashboard/',
 '/usr/share/openstack-dashboard/openstack_dashboard']
Server time: Thu, 19 May 2016 04:26:58 +0000

Revision history for this message
Freerider (seanjohnmcgrath) wrote :

Solution in #6 does not address the problem you are having. Trying running the compression?

You have offline compression enabled but key "337c10dadc650d2900f653d9efa97201" is missing from offline manifest. You may need to run "python manage.py compress".

python manage.py compress

Revision history for this message
Aashish (aash-43) wrote :

Freerider, I'm not executing compression. After adding "WSGIApplicationGroup %{GLOBAL}" in openstack-dashboard.conf file and apache server restart, I started seeing this error related to offline compression.
   When I reverted the changes, I'm getting End of script error as mentioned in #7

Revision history for this message
Aashish (aash-43) wrote :

Freerider, I'm able to access the dashboard now. Thanks for your suggestion.

After solution suggested in #6 I executed "python manage.py compress", the issue was resolved

/usr/share/openstack-dashboard$ sudo python manage.py compress
[sudo] password for controller:
Found 'compress' tags in:
        /usr/share/openstack-dashboard/openstack_dashboard/templates/horizon/_conf.html
        /usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html
        /usr/share/openstack-dashboard/openstack_dashboard/templates/horizon/_scripts.html
Compressing... done
Compressed 4 block(s) from 3 template(s) for 1 context(s).

But is this an additional step for installation in mitaka release, please confirm the same?

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Dashboard (Horizon) because there has been no activity for 60 days.]

Changed in horizon:
status: Incomplete → Expired
Revision history for this message
Praveen N (praveenn) wrote :

Still the issue persists, and after following #6 and #14 resolved for me.

Revision history for this message
maburycn (maburycn) wrote :

I got same situation, even with that 2 resolution, no luck.

Any more suggestions? thanks

Revision history for this message
Maciej Kucia (maciejkucia) wrote :

For anyone with similar issue.

Fix with
 WSGIApplicationGroup %{GLOBAL}
in
 openstack-dashboard.conf
should work.

Please remember to restart httpd (CentOS/Fedora/RHEL).
 systemctl restart httpd

In Ubuntu/Debian
 systemctl start apache2.service

Revision history for this message
Adil Lebbat (adil-lebbat) wrote :

For anyone using Kolla and Kolla-ansible for an allinone or multinode deployment :

- Edit the file /etc/kolla/horizon/horizon.conf on the node/nodes that contain the horizon docker container

After -------> WSGIProcessGroup horizon-http
ADDED THIS -------> WSGIApplicationGroup %{GLOBAL}

- docker stop/start ALL_HORIZON_CONTAINERS_ON_EACH_NODE

This should fix the issue (Kolla 5.0.0 / CentOS 7.4)

Revision history for this message
LIU Yulong (dragon889) wrote :

Queens still has such issue.

Env:
CentOS 7.2
kernel: 3.10.0-514.el7.x86_64
rpm -qa|grep dashboard
openstack-dashboard-13.0.1-1.el7.noarch

Install doc:
https://docs.openstack.org/horizon/queens/install/install-rdo.html

And yes, add the fix to /etc/httpd/conf.d/openstack-dashboard.conf
WSGIApplicationGroup %{GLOBAL}
then restart the httpd service:
systemctl restart httpd.service

Revision history for this message
Akihiro Motoki (amotoki) wrote :

Apart from the documention bug, this is a packaging bug in CentOS package. The horizon upstream development team does not maintain individual package, so a packaging bug needs to be filed to CentOS package.

Note that documentation bug is handled as bug 1741354 and a backport to queens documentation has been proposed.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.