Login to Horizon dashboard fails with "An unexpected error has occurred. Try refreshing the page."

[Sandeep Raman]

OS: Ubuntu Precise, Ubuntu Raring
Horizon version: 1:2013.1+git201303012135~raring-0ubuntu1

Problem Description:
Login to dashboard fails with "An unexpected error has occurred. Try refreshing the page."

From the Github Horizon page, https://github.com/openstack/horizon/blob/master/openstack_dashboard/templates/500.html,
is load_config or horizon_config being called which in turn is missing and causing the issue?

Nothing related to this in the apache2 error log.

[Sandeep Raman (sandeep-raman)]

Adding trace from apache error log:

[Tue Mar 05 06:25:21 2013] [error] \x1b[31;1mUnauthorized: Unauthorized (HTTP 401)\x1b[0m
[Tue Mar 05 06:25:21 2013] [error] Traceback (most recent call last):
[Tue Mar 05 06:25:21 2013] [error] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 112, in get_quotas
[Tue Mar 05 06:25:21 2013] [error] self.quotas = quotas.tenant_quota_usages(self.request)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/horizon/utils/memoized.py", line 33, in __call__
[Tue Mar 05 06:25:21 2013] [error] value = self.func(*args)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 84, in tenant_quota_usages
[Tue Mar 05 06:25:21 2013] [error] for quota in get_tenant_quota_data(request, disabled_quotas):
[Tue Mar 05 06:25:21 2013] [error] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 73, in get_tenant_quota_data
[Tue Mar 05 06:25:21 2013] [error] return _get_quota_data(request, "tenant_quota_get", disabled_quotas)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 61, in _get_quota_data
[Tue Mar 05 06:25:21 2013] [error] quotasets.append(getattr(cinder, method_name)(request, tenant_id))
[Tue Mar 05 06:25:21 2013] [error] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/cinder.py", line 125, in tenant_quota_get
[Tue Mar 05 06:25:21 2013] [error] return QuotaSet(c_client.quotas.get(tenant_id))
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/v1/quotas.py", line 37, in get
[Tue Mar 05 06:25:21 2013] [error] return self._get("/os-quota-sets/%s" % (tenant_id), "quota_set")
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 141, in _get
[Tue Mar 05 06:25:21 2013] [error] resp, body = self.api.client.get(url)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 185, in get
[Tue Mar 05 06:25:21 2013] [error] return self._cs_request(url, 'GET', **kwargs)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 147, in _cs_request
[Tue Mar 05 06:25:21 2013] [error] self.authenticate()
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 293, in authenticate
[Tue Mar 05 06:25:21 2013] [error] auth_url = self._v1_auth(auth_url)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 311, in _v1_auth
[Tue Mar 05 06:25:21 2013] [error] resp, body = self.request(url, 'GET', headers=headers)
[Tue Mar 05 06:25:21 2013] [error] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 136, in request
[Tue Mar 05 06:25:21 2013] [error] raise exceptions.from_response(resp, body)
[Tue Mar 05 06:25:21 2013] [err...

[Gabriel Hurley (gabriel-hurley)]

My guess is that this *is* due to Cinder not being configured but being in your service catalog. If you're not running Cinder you should make sure it's not present as a service in Keystone's service catalog, otherwise you'll get errors like this when trying to retrieve usage information from it in various places.

[Sandeep Raman (sandeep-raman)]

cinder issue is related to iscsitarget compile issue filed at https://bugs.launchpad.net/horizon/+bug/1143897
Could it be an enhancement the UI should display a more meaningful relevant message rather than a generic one?

Other than this there is nothing more in this thread.

[Sandeep Raman (sandeep-raman)]

I did a fresh install of Precise and Grizzly via the ppa repository today. All service is running this time. However the dashboard seems to have the same error reported above. There is no error in the apache log. This needs to be looked into now as it seems to be an issue with dashboard.

[Julie Pichon (jpichon)]

Without an error message it's difficult to debug. If there was an internal server error it should show in the logs. Can you try enabling DEBUG in the local_settings.py file, restarting Apache and trying again? Does anything interesting show up in the apache logs? Can you see in the keystone logs if the request goes through and keystone sends you back a token? Can you access another page directly, e.g. /project/images_and_snapshots/?

The dashboard works fine for many people working on it every day, so it seems more likely to be configuration-related. Maybe someone more familiar with Ubuntu can also chime in about the status of the PPA.

[Sandeep Raman (sandeep-raman)]

After enabling debug, it threw loads of messages. Attached error.log.

Excerpt from the error.log

[Tue Mar 12 03:28:02 2013] [error] File "/usr/lib/python2.7/dist-packages/compressor/templatetags/compress.py", line 81, in render
[Tue Mar 12 03:28:02 2013] [error] cached_offline = self.render_offline(compressor, forced)
[Tue Mar 12 03:28:02 2013] [error] File "/usr/lib/python2.7/dist-packages/compressor/templatetags/compress.py", line 56, in render_offline
[Tue Mar 12 03:28:02 2013] [error] raise OfflineGenerationError('You have offline compression enabled but key "%s" is missing from offline manifest. You may need to run "python manage.py compress".' % key)
[Tue Mar 12 03:28:02 2013] [error] OfflineGenerationError: You have offline compression enabled but key "8d04d8326e2e8769c59d083ce0bb9657" is missing from offline manifest. You may need to run "python manage.py compress".

Executing python manage.py, compress.py and compress.pyc results as below.

root@grizzly:/var/log/apache2# cd /usr/lib/python2.7/dist-packages/compressor/templatetags/
root@grizzly:/usr/lib/python2.7/dist-packages/compressor/templatetags# python manage.py compress
python: can't open file 'manage.py': [Errno 2] No such file or directory
root@grizzly:/usr/lib/python2.7/dist-packages/compressor/templatetags# ls
compress.py compress.pyc __init__.py __init__.pyc
root@grizzly:/usr/lib/python2.7/dist-packages/compressor/templatetags# python compress.py compress
Traceback (most recent call last):
  File "compress.py", line 4, in <module>
    from compressor.cache import (cache_get, cache_set, get_offline_hexdigest,
  File "/usr/lib/python2.7/dist-packages/compressor/cache.py", line 5, in <module>
    from django.core.cache import get_cache
  File "/usr/lib/python2.7/dist-packages/django/core/cache/__init__.py", line 76, in <module>
    if not settings.CACHES:
  File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 184, in inner
    self._setup()
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 40, in _setup
    raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
root@grizzly:/usr/lib/python2.7/dist-packages/compressor/templatetags# python compress.pyc compress
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/compressor/templatetags/compress.py", line 4, in <module>
    from compressor.cache import (cache_get, cache_set, get_offline_hexdigest,
  File "/usr/lib/python2.7/dist-packages/compressor/cache.py", line 5, in <module>
    from django.core.cache import get_cache
  File "/usr/lib/python2.7/dist-packages/django/core/cache/__init__.py", line 76, in <module>
    if not settings.CACHES:
  File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 184, in inner
    self._setup()
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 40, in _setup
    raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cann...

[Julie Pichon (jpichon)]

I'm guessing offline compression was done for the package, which unfortunately often ends up swallowing useful error messages. With DEBUG=True, you should see another traceback inside your browser instead of the "Something unexpected happened" page -- can you paste it in a comment?

[Sandeep Raman (sandeep-raman)]

Sure - http://paste.openstack.org/show/33437/

Btw, I'm able to launch instance from cli. It gets dhcp assigned ip. I'm able to ping and ssh to the instance from the controller node. And also the vnc console for the instance works from my laptop :-)

[Sandeep Raman (sandeep-raman)]

I did another setup today and i see the same error with dashboard.

[Julie Pichon (jpichon)]

Maybe something else worth checking: what happens if you remove the tags {% compress js %} and {% endcompress %} from https://github.com/openstack/horizon/blob/master/horizon/templates/horizon/_scripts.html ?

[Kieran Spear (kspear)]

You can also put:

COMPRESS_ENABLED = False

in your local_settings.py to disable compression altogether.

[Sandeep Raman (sandeep-raman)]

I do not understand how removing the tags from github horizon script will help. Maybe there is something more to be done after removing them from script?

In the local_setting.py, added these lines, no diff.

COMPRESS_OFFLINE = False
COMPRESS_ENABLED = False

[Julie Pichon (jpichon)]

Sandeep: I meant changing these in your local copy of _scripts.html, wherever it was installed. "locate _scripts.html" might help with finding the file location.

Kieran: I saw this on another deployment yesterday and removing the tags helped. If this fixes the problem for the reporter as well, I wonder if something could be interfering with offline compression in Grizzly.

[Matthias Runge (mrunge)]

Sandeep, if you want to compress the offline contents, you should change to the dir, where the dashboard is stored. I'm not into Ubuntu, e.g. in Fedora, you'll find that directory at /usr/share/openstack-dashboard

There you execute ./manage.py compress

If that doesn't change anything, there is another error and the best way to figure out, what's happening, is to set DEBUG=True, COMPRESS_OFFLINE=False, COMPRESS=True

Please verify, you really have cinder either disabled in keystone(!) or have all cinder services up and running, esp. cinder-api!

[Sandeep Raman (sandeep-raman)]

Julie, Matthias,

RC1 continues to show the error.

1. Julie - Removing the tags as suggested allows login.
2. Matthias - I guess Cinder is not disabled in keystone, because i need to use Cinder [how to verify this]. All cinder services is up and running.

./manage.py compress errors as follows -

root@grz-pki1:/usr/share/openstack-dashboard# ./manage.py compress
Found 'compress' tags in:
        /usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html
        /usr/lib/python2.7/dist-packages/horizon/templates/horizon/_conf.html
        /usr/lib/python2.7/dist-packages/horizon/templates/horizon/_scripts.html
Compressing... Error: An error occured during rendering /usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html: /bin/sh: 1: /usr/share/openstack-dashboard/bin/less/lessc: not found

After enabling debug and compress as you suggested, http://10.1.0.44/horizon errors as http://paste.openstack.org/show/34033/

Please help.

[Sandeep Raman (sandeep-raman)]

This combination works -
1. Retain the Ubuntu theme
2. Set COMPRESS_OFFLINE = False in /etc/openstack-dashboard/local_settings.py

So its still a problem with the default OpenStack theme.

[Matthias Runge (mrunge)]

Well, I'd recommend to file a bug against the ubuntu package, IMHO that's a packaging bug.

[Sandeep Raman (sandeep-raman)]

Changed Target to <<Ubuntu--openstack-dashboard>>. Hope that's the right one.

[Sandeep Raman (sandeep-raman)]

After the fresh install, with the Ubuntu theme and COMPRESS_OFFLINE = False, horizon login works.
Disable Ubuntu theme, restart apache and memcached services. Default OpenStack theme login works.
Change back to Ubuntu theme, restart apache2 and memcached service, horizon fails with 500 Internal server error.

[Launchpad Janitor (janitor)]

[Expired for openstack-dashboard (Ubuntu) because there has been no activity for 60 days.]

[Vysakh Chandran (vysakh)]

I got hit by the same issue and no luck with the below workaround . its giving me internal server error.

This combination works -
1. Retain the Ubuntu theme
2. Set COMPRESS_OFFLINE = False in /etc/openstack-dashboard/local_settings.py