Websso fails when HTTP_REFERRER that horizon is unable to connect to gets used

[Georgina Shippey]

I am currently having an issue where a request to Horizon's websso endpoint fails to respond in time as the token validation request fails to connect between Horizon and Keystone.

(On Openstack Train)
I am trying to login to Horizon using an external identity provider.
I have set the WEBSSO_KEYSTONE_URL to keystones external facing endpoint as the IDP is on an external network.

The POST request to https://horizon_ip/auth/websso/ that includes a keystone token for validation in its params is failing.
This request routes to the horizon view 'websso' (https://opendev.org/openstack/horizon/src/branch/master/openstack_auth/views.py#L165)
The token authentication request to keystone in this view uses the requests HTTP_REFERRER when available as the keystone endpoint to use.
The previous request was to keystone on its external endpoint (as used by the external identity provider) to its route 'auth/OS-FEDERATION/websso/openid', and therefore the HTTP_REFERRER for this POST request is the external keystone endpoint.

Our Openstack services have minimal external connectivity for security reasons.
So in our setup the horizon service is unable to make connections to the external keystone endpoint.
Therefore in the horizon apache logs I see:
  Unable to establish connection to https://keystone_external_ip:5000/v3/auth/tokens
Which eventually leads to a time out.

As this is request between Horizon and Keystone ideally for us it should be using the internal endpoint. I've had a go at setting the auth_url to be settings.OPENSTACK_KEYSTONE_URL and this lets me login successfully.

I am unsure as to why the HTTP_REFERRER gets used in preference over the settings.OPENSTACK_KEYSTONE_URL for this request?

I propose either:
1. Removing the use of HTTP_REFERRER in favor of settings.OPENSTACK_KEYSTONE_URL.
2. Providing a setting to toggle between using the HTTP_REFERRER or settings.OPENSTACK_KEYSTONE_URL to build the auth request with.

Original commit in django_openstack_auth for websso view: https://github.com/openstack/django_openstack_auth/commit/302f422568a32b513ffbb3089ba799a4416df108

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/722685

[Jeff Albert (jralbert)]

This affects our installation as well, and will affect anyone running an OpenStack environment in which the control plane doesn't have outbound network connectivity to hit public endpoints, and instead relies on internal endpoints on a shared network. This is a somewhat common security control to implement, and the change the original bug reporter has proposed seems both effective and narrowly-scoped so that it should be a fairly simple thing to review. Can the proposed fix be merged?

[Jonathan Rosser (jrosser)]

With a control plane that has no outbound connectivity it's very important that the correct endpoints can be used/specified.

The horizon service itself must talk to keystone on the internal endpoint, whilst the communication between the end user browser, the IdP, horizon, and the public openstack APIs only happen via the external endpoint.

The use of HTTP_REFERRER as described in the bug report mixes the internal and external physical network contexts in a way which leads to websso being broken for some deployments, but it will work for others with less strict (or no... devstack?) isolation.

We are currently having to maintain a fork of horizon with https://review.opendev.org/722685 applied in order to get websso to work with a strictly segregated network. Comment #2 shows this is not a unique situation.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/722685
Committed: https://opendev.org/openstack/horizon/commit/33292ca0a467637971c73f420166b4077e941e20
Submitter: "Zuul (22348)"
Branch: master

commit 33292ca0a467637971c73f420166b4077e941e20
Author: Georgina Shippey <email address hidden>
Date: Fri Apr 24 13:52:42 2020 +0100

    Use OPENSTACK_KEYSTONE_URL instead of HTTP_REFERRER

    By using OPENSTACK_KEYSTONE_URL instead of the HTTP_REFERRER
    the authentication request between Horizon and Keystone continues
    to work in situations where the HTTP_REFERRER is an external keystone
    endpoint that Horizon does not have access to.

    Change-Id: I9c5c8d59c5f5a8570dbb563ae224d45406a73ba5
    Closes-bug: #1874705

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 21.0.0 release.