gate/check-grenade-dsvm: Horizon front page not functioning!

[Chmouel Boudjnah]

Grenade dsvm fails with horizon :

2013-12-30 16:15:32.294 | + [[ ,g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,sysstat,s-proxy,s-account,s-container,s-object,cinder,c-api,c-vol,c-sch,n-cond,tempest,c-bak,n-net, =~ ,horizon, ]]
2013-12-30 16:15:32.294 | + return 0
2013-12-30 16:15:32.295 | + curl <a href='http://127.0.0.1'>http://127.0.0.1</a>
2013-12-30 16:15:32.295 | + grep -q '&lt;h3>Log In&lt;/h3>'
2013-12-30 16:15:32.337 | + die 39 'Horizon front page not functioning!'
2013-12-30 16:15:32.338 | + local exitcode=1
2013-12-30 16:15:32.339 | + set +o xtrace
2013-12-30 16:15:32.339 | [Call Trace]
2013-12-30 16:15:32.377 | /opt/stack/new/devstack/exercises/horizon.sh:39:die
2013-12-30 16:15:32.379 | [ERROR] /opt/stack/new/devstack/exercises/horizon.sh:39 Horizon front page not functioning!

example review where it's failing https://review.openstack.org/#/c/60707

[Davanum Srinivas (DIMS) (dims-v)]

Looks like it cropped up just this morning - log stash query is:
"[ERROR] /opt/stack/new/devstack/exercises/horizon.sh:39 Horizon front page not functioning!"

http://logstash.openstack.org/#eyJzZWFyY2giOiJcIltFUlJPUl0gL29wdC9zdGFjay9uZXcvZGV2c3RhY2svZXhlcmNpc2VzL2hvcml6b24uc2g6MzkgSG9yaXpvbiBmcm9udCBwYWdlIG5vdCBmdW5jdGlvbmluZyFcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4ODQ0MzIxMzU2N30=

[Davanum Srinivas (DIMS) (dims-v)]

gate-grenade-dsvm jobs are failing as well

[Joe Gordon (jogo)]

This patch doesn't halt the screen session, so old/horizon.log should say what is going wrong.

I have a hunch the problem here is related to the fact that grenade doesn't upgrade horizon at all -- but we try testing it.

[Joe Gordon (jogo)]

https://review.openstack.org/#/c/63965/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/64491

[Davanum Srinivas (DIMS) (dims-v)]

Joe,

I found this in the apache2/horizon log. (http://logs.openstack.org/90/64490/1/check/check-grenade-dsvm/60ce30b/logs/horizon_error.txt.gz) will the review above help with this?

[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] mod_wsgi (pid=16849): Target WSGI script '/opt/stack/old/horizon/openstack_dashboard/wsgi/django.wsgi' cannot be loaded as Python module.
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] mod_wsgi (pid=16849): Exception occurred processing WSGI script '/opt/stack/old/horizon/openstack_dashboard/wsgi/django.wsgi'.
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] Traceback (most recent call last):
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] File "/opt/stack/old/horizon/openstack_dashboard/wsgi/django.wsgi", line 4, in <module>
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] import django.core.handlers.wsgi
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 9, in <module>
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] from django import http
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] File "/usr/local/lib/python2.7/dist-packages/django/http/__init__.py", line 4, in <module>
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] from django.http.response import (HttpResponse, StreamingHttpResponse,
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] File "/usr/local/lib/python2.7/dist-packages/django/http/response.py", line 17, in <module>
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] from django.utils import six, timezone
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] File "/usr/local/lib/python2.7/dist-packages/django/utils/timezone.py", line 11, in <module>
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] import pytz
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] File "/usr/local/lib/python2.7/dist-packages/pytz/__init__.py", line 29, in <module>
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] from pkg_resources import resource_stream
[Tue Dec 31 02:26:43 2013] [error] [client 127.0.0.1] IOError: zipimport: can not open file /usr/local/lib/python2.7/dist-packages/setuptools-2.0.1-py2.7.egg

[Joe Gordon (jogo)]

Dims stacktrace confirms the theory is related to grenade not upgrading horizon. we upgrade all the python dependencies that horizon is using but don't upgrade horizon, so if it tries to import something upon receiving a GET request, it can fail because the wrong dependency is now installed.

[Davanum Srinivas (DIMS) (dims-v)]

This bug has reared up again

http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkhvcml6b24gZnJvbnQgcGFnZSBub3QgZnVuY3Rpb25pbmchXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODkxOTQwMjc4NzB9

[Davanum Srinivas (DIMS) (dims-v)]

114 hits in last 12 hours

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/65489

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65489
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=63062df64e55e137380b762de3b6fb51995b7473
Submitter: Jenkins
Branch: master

commit 63062df64e55e137380b762de3b6fb51995b7473
Author: Matt Riedemann <email address hidden>
Date: Wed Jan 8 07:50:33 2014 -0800

    Add query for bug 1265057

    Related-Bug: #1265057

    Change-Id: Ic1c53386e3991e56d534daba876b97a21a563b3f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/64491
Committed: https://git.openstack.org/cgit/openstack-dev/grenade/commit/?id=e9546e66e6236a1e2c79734ee0c9dcb0cc50bbb3
Submitter: Jenkins
Branch: master

commit e9546e66e6236a1e2c79734ee0c9dcb0cc50bbb3
Author: Joe Gordon <email address hidden>
Date: Mon Dec 30 18:26:06 2013 -0800

    Add upgrade-horizon

    Upgrade horizon along with the rest of devstack before running new
    devstack tests and tempest.

    upgrade-horizon is based off of upgrade-keystone.

    Change-Id: I6a6e4da06806cb840d7af673673e06f34eb297e0
    Related-Bug: #1265057

[Alan Pevec (apevec)]

https://review.openstack.org/64491 needs to be backported to grenade stable/havana too!

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/71731

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/71731
Committed: https://git.openstack.org/cgit/openstack-dev/grenade/commit/?id=97fea9b26799b6b4a6f951af9d3cf8f75f7c00e9
Submitter: Jenkins
Branch: stable/havana

commit 97fea9b26799b6b4a6f951af9d3cf8f75f7c00e9
Author: Joe Gordon <email address hidden>
Date: Mon Dec 30 18:26:06 2013 -0800

    Add upgrade-horizon

    Upgrade horizon along with the rest of devstack before running new
    devstack tests and tempest.

    upgrade-horizon is based off of upgrade-keystone.

    Change-Id: I6a6e4da06806cb840d7af673673e06f34eb297e0
    Related-Bug: #1265057
    (cherry picked from commit e9546e66e6236a1e2c79734ee0c9dcb0cc50bbb3)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/101931

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/101931
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=3c6281a7d4884d2b291cb00d8a7358a1c6d1da96
Submitter: Jenkins
Branch: master

commit 3c6281a7d4884d2b291cb00d8a7358a1c6d1da96
Author: Sean Dague <email address hidden>
Date: Mon Jun 23 11:18:29 2014 -0400

    this query is too broad to be useful

    And we have better queries now that we are indexing horizon_error.log

    Change-Id: I2eac509325186236a4f3a7a6ace45eacce563eed
    Related-Bug: #1265057

[Matt Riedemann (mriedem)]

This is blowing up again:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiKyBkaWUgMzkgJ0hvcml6b24gZnJvbnQgcGFnZSBub3QgZnVuY3Rpb25pbmchJ1wiIEFORCB0YWdzOlwiY29uc29sZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI2NTk3NDAxOTU3fQ==

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/165059

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/165059
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=e0f1816c1e57372e155e1c490c5a27436932cd30
Submitter: Jenkins
Branch: master

commit e0f1816c1e57372e155e1c490c5a27436932cd30
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 17 13:06:19 2015 +0000

    Restore query for horizon bug 1265057

    This reverts commit 3c6281a7d4884d2b291cb00d8a7358a1c6d1da96

    The bug is back and there is nothing in horizon_error.log.

    Change-Id: Id80848fbc55eec2d9ae9416d698aadb7d5282e0f
    Related-Bug: #1265057

[Matt Riedemann (mriedem)]

This is removing the horizon exercise from devstack: https://review.openstack.org/#/c/165063/

[Doug Fish (drfish)]

The exercise failed because Horizon was updated to redirect to the initial login page rather than GET it directly. curl needs the -L flag in order to follow redirects.

I haven't put this back in because I'm confused about how the first patch passed the gate without failure. I'd like to understand that before re-introducing the exercise.

[Matt Riedemann (mriedem)]

The horizon exercises shouldn't come back for checking the login page, we have a test for that in tempest:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_dashboard_basic_ops.py#n70

[Sean Dague (sdague)]

This devstack bug was last updated over 180 days ago, as devstack
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.