Keystone 403 Forbidden

[Tiago Ferreira]

Hello there,
I have been struggling a bit with moving the horizon page from domain.com/horizon to domain.com/ + setting up HTTPS, here's what I have tied:

 - [General 403 debugging](https://askubuntu.com/questions/292968/apache2-forbidden-you-dont-have-permission-to-access-dir-on-this-server)
 - [General 403 debugging](https://unix.stackexchange.com/questions/169513/403-forbidden-you-dont-have-permission-to-access-on-this-server-apache2)
 - [403 fix for Horizon](https://fosshelp.blogspot.co.uk/2014/02/openstack-horizon-you-dont-have.html)
 - [General 403 debugging](https://stackoverflow.com/questions/10873295/error-message-forbidden-you-dont-have-permission-to-access-on-this-server)
 - [Launchpad 403 knwon (and fixed) bug ](https://bugs.launchpad.net/devstack/+bug/1243075)
 - [HTTPS config guide (from Juno)](https://docs.openstack.org/juno/config-reference/content/configure-dashboard.html#after-example)

### Environment
 - Followed the latest installation guide (Ocata)
 - Apache2 version: Apache/2.4.18 (Ubuntu)
 - Ubuntu 16.04.2 LTS AMD64

### Configuration

**local_settings.py**
https://paste.debian.net/930199/

**openstack-dashboard.conf**
https://paste.debian.net/930200/

**error.log**
https://paste.debian.net/930201/

This leaves me with some funny results, the login page loads but is missing CSS, only plain HTML. When I log in, everything else gives me a `403`.

Any help would be appreciated.

Thank you,

Tiago Ferreira

<email address hidden>

[David Lyle (david-lyle)]

Tiago, I think the problem may actually lie in your webserver (Apache) config. Typically, horizon.conf maps the static directory as well as the root content. Wondering if when you changed the website root, you overlooked the static mapping. See https://github.com/openstack-dev/devstack/blob/master/files/apache-horizon.template#L10 as an example

[Tiago Ferreira (tiferrei)]

I'm sorry David but this is a bit new for me, I have added the DocumentRoot but how will that fix the Forbidden errors?

Thanks,
Tiago

[David Lyle (david-lyle)]

Lines 11 &

[David Lyle (david-lyle)]

Lines 11 & 12 were the important ones in that block. You need to specify the URL path for static and media files. where %WEBROOT% matches what you have for %WEBROOT% on the WSGIScriptAlias line.

[Tiago Ferreira (tiferrei)]

Hi David,
I'm sorry but I don't see how these 3 lines can "coexist":

`DocumentRoot %HORIZON_DIR%/.blackhole/`
`Alias %WEBROOT%/media %HORIZON_DIR%/openstack_dashboard/static`
`Alias %WEBROOT%/static %HORIZON_DIR%/static`

Firstly because my Horizon dir is `/usr/share/openstack-dashboard` and in `/usr/share/openstack-dashboard` there is no `static` directory. Only in `/var/lib/openstack-dashboard`.
But then in `/var/lib/openstack-dashboard` there is also no `openstack_dashboard` directory.
So my horizon dir can't be either of them?

[David Lyle (david-lyle)]

The media line is less important as Horizon doesn't use that by default, but the line is added to be complete as media is a support Django top level content directory. The directory you want is the directory that results from running 'python manage.py collectstatic' and 'python manage.py compress'. You're just mapping the directory with the generated static files to a url. The contents of static should have a couple <uuid>.js and <uuid>.css files in it.

[Tiago Ferreira (tiferrei)]

Sorry, nothing :(
https://paste.debian.net/930497/

Also static only seems to have folders, no files:
total 40
drwxr-xr-x 10 www-data www-data 4096 Apr 30 23:13 .
drwxr-xr-x 4 www-data www-data 4096 Apr 30 23:13 ..
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 app
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 auth
drwxr-xr-x 8 www-data www-data 4096 Apr 30 23:15 dashboard
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 framework
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 horizon
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 js
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 scss
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 themes

[David Lyle (david-lyle)]

that is the wrong static, you need one level above that. The base directory where horizon is installed will have a static directory with files in the naming convention I mentioned before. So you're looking for horizon/static not horizon/openstack_dashboard/static or horizon/horizon/static.

[Tiago Ferreira (tiferrei)]

OK so I found the correct directory: /usr/share/openstack-dashboard/horizon/static, however this does not contain the files either, just folders:
total 20
drwxr-xr-x 5 root root 4096 Apr 30 23:00 .
drwxr-xr-x 19 root root 4096 Apr 30 23:01 ..
drwxr-xr-x 3 root root 4096 Apr 30 23:01 auth
drwxr-xr-x 5 root root 4096 Apr 30 23:01 framework
drwxr-xr-x 5 root root 4096 Apr 30 23:01 horizon

[David Lyle (david-lyle)]

That's not right one either. The typical structure of the source tree on a deployed system is as you see it in https://github.com/openstack/horizon and it's at this very top level that a static directory gets created.

[Tiago Ferreira (tiferrei)]

Bloody hell, how many openstack-dashboard directories are there after all?

These are the directories I have found so far:

ls -la /usr/share/openstack-dashboard
total 52
drwxr-xr-x 3 root root 4096 Apr 30 23:15 .
drwxr-xr-x 132 root root 4096 Apr 30 23:13 ..
lrwxrwxrwx 1 root root 41 Apr 2 11:44 horizon -> ../../lib/python2.7/dist-packages/horizon
-rwxr-xr-x 1 root root 886 Apr 2 11:44 manage.py
-rw-r--r-- 1 root root 508 Apr 30 23:15 manage.pyc
drwxr-xr-x 19 www-data www-data 4096 Apr 30 23:15 openstack_dashboard
-rw-r--r-- 1 root root 15180 Apr 2 11:44 settings.py
-rw-r--r-- 1 root root 13181 Apr 30 23:15 settings.pyc

ls -la /var/lib/openstack-dashboard/
total 20
drwxr-xr-x 4 www-data www-data 4096 Apr 30 23:13 .
drwxr-xr-x 50 root root 4096 Apr 30 23:13 ..
-rw------- 1 www-data www-data 64 Apr 30 23:13 secret_key
drwxr-xr-x 2 www-data www-data 4096 Apr 2 11:44 secret-key
drwxr-xr-x 10 www-data www-data 4096 Apr 30 23:13 static
-rw-r--r-- 1 www-data www-data 0 Apr 30 23:13 _var_lib_openstack-dashboard_secret_key.lock

ls -la /etc/openstack-dashboard/
total 40
drwxr-xr-x 2 horizon horizon 4096 May 1 12:34 .
drwxr-xr-x 107 root root 4096 May 3 04:05 ..
-rw-r--r-- 1 root root 32158 May 1 12:34 local_settings.py

Two of them contain static folders:

ls -la /usr/share/openstack-dashboard/openstack_dashboard/static
total 20
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 .
drwxr-xr-x 19 www-data www-data 4096 Apr 30 23:15 ..
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 app
drwxr-xr-x 4 www-data www-data 4096 Apr 30 23:13 dashboard
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 js

ls -la /usr/share/openstack-dashboard/horizon/static
total 20
drwxr-xr-x 5 root root 4096 Apr 30 23:00 .
drwxr-xr-x 19 root root 4096 Apr 30 23:01 ..
drwxr-xr-x 3 root root 4096 Apr 30 23:01 auth
drwxr-xr-x 5 root root 4096 Apr 30 23:01 framework
drwxr-xr-x 5 root root 4096 Apr 30 23:01 horizon

ls -la /var/lib/openstack-dashboard/static
total 40
drwxr-xr-x 10 www-data www-data 4096 Apr 30 23:13 .
drwxr-xr-x 4 www-data www-data 4096 Apr 30 23:13 ..
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 app
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 auth
drwxr-xr-x 8 www-data www-data 4096 Apr 30 23:15 dashboard
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 framework
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 horizon
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 js
drwxr-xr-x 3 www-data www-data 4096 Apr 30 23:13 scss
drwxr-xr-x 5 www-data www-data 4096 Apr 30 23:13 themes

Am I missing another hidden openstack directory?
I am sorry if I'm missing something very obvious but why do we even have 3 directories?

[David Lyle (david-lyle)]

what system are you on that everything is names openstack_dashboard? my guess is /var/lib/openstack-dashboard/static may be the correct directory, although

lrwxrwxrwx 1 root root 41 Apr 2 11:44 horizon -> ../../lib/python2.7/dist-packages/horizon

looks promising as well.

[Tiago Ferreira (tiferrei)]

Neither of them worked :(

[Tiago Ferreira (tiferrei)]

I'm on Ubuntu Server 16.04.2 LTS

[David Lyle (david-lyle)]

how did you set up horizon, with Ubuntu packages? You may want to ask this question in the launchpad project for the horizon ubuntu packages. I'm not sure the intended location or flow for those packages. But essentially, assuming you have offline compression turned on you would need to run "python manage.py collectstatic" and "python manage compress" from the base horizon directory to populate the static directory necessary. But I can only provide guidance on how to do it from source.

If you wanted to test things out, you could clone the repo and run the above two python commands then update your apache horizon.conf to point to these new directories to get a better understand of how and where things are happening.

[Tiago Ferreira (tiferrei)]

I just installed horizon with Apt, the package is called openstack-dashboard. Set some settings on the dashboard config files and it's ready to go (there's no documentation about https or moving the dashboard to the domain root)

Thank you, Tiago

On 4 May 2017, 00:05 +0100, David Lyle <email address hidden>, wrote:
> how did you set up horizon, with Ubuntu packages? You may want to ask
> this question in the launchpad project for the horizon ubuntu packages.
> I'm not sure the intended location or flow for those packages. But
> essentially, assuming you have offline compression turned on you would
> need to run "python manage.py collectstatic" and "python manage
> compress" from the base horizon directory to populate the static
> directory necessary. But I can only provide guidance on how to do it
> from source.
>
> If you wanted to test things out, you could clone the repo and run the
> above two python commands then update your apache horizon.conf to point
> to these new directories to get a better understand of how and where
> things are happening.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1687401
>
> Title:
> Horizon 403 Forbidden
>
> Status in OpenStack Dashboard (Horizon):
> Incomplete
>
> Bug description:
> Hello there,
> I have been struggling a bit with moving the horizon page from domain.com/horizon to domain.com/ + setting up HTTPS, here's what I have tied:
>
> - [General 403 debugging](https://askubuntu.com/questions/292968/apache2-forbidden-you-dont-have-permission-to-access-dir-on-this-server)
> - [General 403 debugging](https://unix.stackexchange.com/questions/169513/403-forbidden-you-dont-have-permission-to-access-on-this-server-apache2)
> - [403 fix for Horizon](https://fosshelp.blogspot.co.uk/2014/02/openstack-horizon-you-dont-have.html)
> - [General 403 debugging](https://stackoverflow.com/questions/10873295/error-message-forbidden-you-dont-have-permission-to-access-on-this-server)
> - [Launchpad 403 knwon (and fixed) bug ](https://bugs.launchpad.net/devstack/+bug/1243075)
> - [HTTPS config guide (from Juno)](https://docs.openstack.org/juno/config-reference/content/configure-dashboard.html#after-example)
>
> ### Environment
> - Followed the latest installation guide (Ocata)
> - Apache2 version: Apache/2.4.18 (Ubuntu)
> - Ubuntu 16.04.2 LTS AMD64
>
> ### Configuration
>
> **local_settings.py**
> https://paste.debian.net/930199/
>
> **openstack-dashboard.conf**
> https://paste.debian.net/930200/
>
> **error.log**
> https://paste.debian.net/930201/
>
> This leaves me with some funny results, the login page loads but is
> missing CSS, only plain HTML. When I log in, everything else gives me
> a `403`.
>
> Any help would be appreciated.
>
> Thank you,
>
> Tiago Ferreira
>
> <email address hidden>
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/horizon/+bug/1687401/+subscriptions

[David Lyle (david-lyle)]

/usr/share/openstack-dashboard/static seems to be the location in http://packages.ubuntu.com/xenial/all/openstack-dashboard/filelist

[David Lyle (david-lyle)]

reviewing you pasted local_settings.py, I see COMPRESS_OFFLINE = True, so either the ubuntu package has these pre-populated, or you'll have to run the two python manage.py commands I have mentioned in the /usr/share/openstack-dashboard directory to populate that static directory.

[Tiago Ferreira (tiferrei)]

Hum now the openstack logo doesn't even load, only pure html text...

When I run the commands I get:
You have requested to collect static files at the destination
location as specified in your settings:

    /var/lib/openstack-dashboard/static

This will overwrite existing files!
Are you sure you want to do this?

Is it possible that /var/lib/openstack-dashboard/static is the location?
But again, /var/lib/openstack-dashboard/static has no files, only dirs and /usr/share/openstack-dashboard still has no static folder, even after running the commands successfully.

Also this might help: /usr/share/openstack-dashboard/settings.py
https://paste.debian.net/930783/

[Tiago Ferreira (tiferrei)]

Also, am I the only one finding it interesting that /var/lib/openstack-dashboard isn't listed in the package file list?

[David Lyle (david-lyle)]

/var/lib/openstack-dashboard/static is the STATIC_ROOT listed in your settings.py, so that should be the target. Do you have permission to write to /var/lib/openstack-dashboard/static?

[Tiago Ferreira (tiferrei)]

I am root :)
OK so I have reinstalled everything. At the moment my apache conf is standard, I have configured local-settings but without ssl, and everything works. So how about we start things from scratch, first we should take a look at moving the webroot. However, my webroot is already "/" on local_settings, so I don't know what else to change. Everything is still under /horizon/ though.

[Tiago Ferreira (tiferrei)]

openstack-dashboard.conf:

WSGIScriptAlias /horizon /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi process-group=horizon
WSGIDaemonProcess horizon user=www-data group=www-data processes=3 threads=10 display-name=%{GROUP}
WSGIProcessGroup horizon

Alias /static /var/lib/openstack-dashboard/static/
Alias /horizon/static /var/lib/openstack-dashboard/static/

<Directory /usr/share/openstack-dashboard/openstack_dashboard/wsgi>
  Require all granted
</Directory>

<Directory /var/lib/openstack-dashboard/static>
  Require all granted
</Directory>

Funny enough, the Alias in apache already comes with the root as /static, but if I remove /horizon/static, I get the 403s

[Tiago Ferreira (tiferrei)]

OK so now with:
WSGIScriptAlias / /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi process-group=horizon
WSGIDaemonProcess horizon user=www-data group=www-data processes=3 threads=10 display-name=%{GROUP}
WSGIProcessGroup horizon

Alias /static /var/lib/openstack-dashboard/static/
#Alias /horizon/static /var/lib/openstack-dashboard/static/

<Directory /usr/share/openstack-dashboard/openstack_dashboard/wsgi>
  Require all granted
</Directory>

<Directory /var/lib/openstack-dashboard/static>
  Require all granted
</Directory>

And the python command, the login page works perfectly. But as soon as I login, I am redirected to the correct URL (the domain root instead of /horizon) but the 403 errors kick in.

[David Lyle (david-lyle)]

ok, I have a new theory, "/horizon/static" is hardcoded in one of these

/usr/share/openstack-dashboard-ubuntu-theme/static/themes/ubuntu/_styles.scss
/usr/share/openstack-dashboard-ubuntu-theme/static/themes/ubuntu/_variables.scs

[Tiago Ferreira (tiferrei)]

But the CSS is working now, it's just some pages that give 403s.

[Tiago Ferreira (tiferrei)]

Eureka!! It works now! Most times...
Basically the only page that gives me the 403s is /identity/ which is the one that would be shown after the login, If I login, get the 403, and manually go to domain.com/project or any other page, it works!

[Tiago Ferreira (tiferrei)]

It still redirects me to /identity/ every time I login, and it still gives 403 on that page.

[Tiago Ferreira (tiferrei)]

I believe this is an issue with Keystone, as only the keystone-endpoints give me 403s

[Tiago Ferreira (tiferrei)]

I am unable to communicate with Keystone via the HTTP API, I have tried both internal and admin endpoints but I always get a 401 Unauthorised.
When accessing Keystone properties in the Dashboard (horizon) I also get a 403, however, everything else works fine.

I have also tried manually composing some HTTP requests from the CLI:

curl -vv -X POST -H 'Content-Type: application/json' -d '{
"auth": {
        "identity": {
             "methods": ["password"],
            "password": {
                    "user": {
                      "name": "tiferrei",
                      "domain": { "id": "default" },
                      "password": "<password>"
                   }
              }
        },
        "scope": {
             "project": {
                "name": "default",
                "domain": { "id": "default" }
              }
        }
    }
}' http://controller:5000/v3/auth/tokens ; echo

And:

curl -vv -X POST -H 'Content-Type: application/json' -d '{
"auth": {
        "identity": {
             "methods": ["password"],
            "password": {
                    "user": {
                      "name": "tiferrei",
                      "domain": { "id": "default" },
                      "password": "<password>"
                   }
              }
        },
        "scope": {
             "project": {
                "name": "default",
                "domain": { "id": "default" }
              }
        }
    }
}' http://controller:35357/v3/auth/tokens ; echo

They both return:
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

Any ideas as to why I'm being denied access to Keystone?

[David Lyle (david-lyle)]

Are you running horizon and keystone services on the same system? If so, there will be collisions for certain URLs. Keystone is now served by apache as well and once you remove the /dashboard or /horizon URL prefix for Horizon, the /identity or /identity/projects (can't remember which for certain) has meaning in both Horizon and Keystone. One option to make horizon seem to be served from / and avoid conflicts is to have apache redirect / to /horizon automatically. devstack does this, see this line in the horizon apache config template: https://github.com/openstack-dev/devstack/blob/master/files/apache-horizon.template#L14

[Tiago Ferreira (tiferrei)]

OK so I moved horizon to /dashboard, now the /identity pages work, but everything is missing the CSS.
On the error logs there's also:
[Wed May 17 18:45:13.419744 2017] [wsgi:error] [pid 22322:tid 140000746075904] Not Found: /dashboard/project/dashboard/static/dashboard/img/logo.svg
[Wed May 17 18:45:13.946599 2017] [wsgi:error] [pid 22321:tid 140000746075904] Not Found: /dashboard/project/dashboard/static/dashboard/js/4e158a721cfe.js
[Wed May 17 18:45:13.965944 2017] [wsgi:error] [pid 22321:tid 140000712505088] Not Found: /dashboard/project/dashboard/static/dashboard/css/34f8a8f8d5e5.css
[Wed May 17 18:45:13.980722 2017] [wsgi:error] [pid 22321:tid 140000729290496] Not Found: /dashboard/project/dashboard/static/dashboard/js/28cba46e213f.js
[Wed May 17 18:45:13.987172 2017] [wsgi:error] [pid 22321:tid 140000737683200] Not Found: /dashboard/project/dashboard/static/dashboard/css/8d0fb71080d5.css
[Wed May 17 18:45:14.035611 2017] [wsgi:error] [pid 22321:tid 140000720897792] Not Found: /dashboard/project/dashboard/static/dashboard/js/830e26e34b64.js

And the Keystone requests still don't work, even though I have resolved the conflicts.

[David Lyle (david-lyle)]

the css fix would be to undo some of the steps we did above. I'll leave the keystone question for a keystone dev.

[Tiago Ferreira (tiferrei)]

OK perfect, I was able to get the static content back by playing around a bit with the Aliases and the python compress commands. But the keystone still isn't working correctly. Thank you for helping the beginner, David!

Tiago

[Lance Bragstad (lbragstad)]

Hi Tiago,

Are you able to pull any information from the keystone logs?

[Lance Bragstad (lbragstad)]

Is there any update on the keystone-specific issues?

[Lance Bragstad (lbragstad)]

Marking as Incomplete for now until we get a little more information.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.]