Accessing Horizon fails with a 503 error

[Ben Nemec]

I'm seeing this behavior in two separate local environments. If you try to access Horizon you just get back a 503. So far I haven't been able to find any logs that explain why this is happening. All of the Horizon logs are empty (which may explain the 503, but isn't terribly helpful...).

Worth noting that this also happens if I bypass haproxy and hit the private address directly.

[Ben Nemec (bnemec)]

Okay, figured out the problem. We're missing a Listen directive in ports.conf for port 80, which causes Apache to not listen on the Horizon port. I suppose this is a puppet thing?

This also doesn't seem to have gotten Horizon fully working either. I can get to the login page, but when I log in I get a CSRF failure now. :-(

[Ben Nemec (bnemec)]

Looks like the CSRF problem is that if you deploy without SSL we configure Horizon in a broken way. Details in https://ask.openstack.org/en/question/56838/solaris-112-openstack-csrf-verification-failed/

When I commented out those two options in my Horizon local_settings I was able to get in. We should fix that, although I suspect non-SSL Horizon is a fairly rare use case in the real world. As long as we support non-SSL deployments they should work as expected though.

[Ben Nemec (bnemec)]

I opened https://bugs.launchpad.net/tripleo/+bug/1696861 for the CSRF bug. I still haven't figured out how to fix the missing Listen directive problem though.

[Ben Nemec (bnemec)]

Hmm, for some reason add_listen is false when we call the horizon apache::vhost resource:

2017-06-08 22:40:45 +0000 Puppet (debug): Create new resource apache::vhost[horizon_vhost] with params {"servername"=>"overcloud-controller-0.localdomain", "serveraliases"=>["overcloud-controller-0.localdomain"], "docroot"=>"/var/www/", "access_log_file"=>"horizon_access.log", "error_log_file"=>"horizon_error.log", "priority"=>10, "aliases"=>[{"alias"=>"/dashboard/static", "path"=>"/usr/share/openstack-dashboard/static"}], "port"=>80, "ssl_cert"=>:undef, "ssl_key"=>:undef, "ssl_ca"=>:undef, "wsgi_script_aliases"=>{"/dashboard"=>"/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi"}, "wsgi_daemon_process"=>"apache", "wsgi_daemon_process_options"=>{"processes"=>"3", "threads"=>"10", "user"=>"apache", "group"=>"apache"}, "wsgi_import_script"=>"/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi", "wsgi_process_group"=>"apache", "redirectmatch_status"=>"permanent", "ip"=>"9.1.1.12", "access_log_format"=>"%a %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\"", *****"add_listen"=>false*****, "options"=>["FollowSymLinks", "MultiViews"], "redirectmatch_regexp"=>"^/$", "redirectmatch_dest"=>"/dashboard"}

Not sure why. It defaults to try in the apache module, and I don't see that we're setting it false anywhere. :-/

[Ben Nemec (bnemec)]

Oh, we did this on purpose: https://github.com/openstack/tripleo-heat-templates/blob/2518394c2f45d5514946de962f8dfd13aa7c7377/puppet/services/horizon.yaml#L92

That is...confusing. I'm not sure how this ever worked or why we would have wanted to disable that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/472756

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/472756
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=93b42baf5181cf7ee75d74581e678ba87ea4b2d7
Submitter: Jenkins
Branch: master

commit 93b42baf5181cf7ee75d74581e678ba87ea4b2d7
Author: Ben Nemec <email address hidden>
Date: Fri Jun 9 11:47:50 2017 -0500

    Remove add_listen: false from Horizon hieradata

    I'm not sure why this was here, but without a Listen directive in
    Apache's ports.conf Horizon is inaccessible. Removing this allows
    Horizon to work again.

    Change-Id: Ic221e15f188cf50b485e995035cb96f5d5960a72
    Closes-Bug: 1696439

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b3 development milestone.