Heat API cannot cope with being behind an SSL terminator

[Clint Byrum]

We recently setup heat-api to listen without SSL behind stunnel which was doing SSL. This is sort of like a simulation of what may be a common setup in production, where SSL termination might be done on a specific SSL-only proxy and then forwarded to heat-api unencrypted.

The symptom is that sometimes Heat's API uses 302 redirects to route requests:

When running heat --debug event-list some-stack against a setup like this, the symptom is:

---begin---
[chopped token]
-H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' https://cd-undercloud.tripleo.org:13004//v1/4956c533154c476799c688eda7ed65ab/stacks/overcloud/events

HTTP/1.1 302 Found
date: Sat, 05 Oct 2013 04:34:16 GMT
content-length: 216
content-type: text/plain; charset=UTF-8
location: http://cd-undercloud.tripleo.org:13004//v1/4956c533154c476799c688eda7ed65ab/stacks/overcloud/50b55f6c-4a13-4064-831e-b0ac0bbddaea/events

302 Found

The resource was found at http://cd-undercloud.tripleo.org:13004//v1/4956c533154c476799c688eda7ed65ab/stacks/overcloud/50b55f6c-4a13-4064-831e-b0ac0bbddaea/events; you should be redirected automatically.
---end---

The problem is that Heat is assuming http because it doesn't know about the SSL.

This is a common problem for most backend web application servers, and is usually solved by allowing one to override the scheme/port for redirects in a config option.

[Steve Baker (steve-stevebaker)]

Is this problem limited to redirects, or also to links URLs in REST bodies to other REST resources?

[Thomas Herve (therve)]

 Is the problem specific to Heat? It seems that it should have been solved by other openstack projects already. I don't suppose we're the only ones doing redirect.

[Clint Byrum (clint-fewbar)]

Excerpts from Steve Baker's message of 2013-10-05 19:26:26 UTC:
> Is this problem limited to redirects, or also to links URLs in REST
> bodies to other REST resources?
>

I think this is specific to all non-relative addresses that the Heat
API serves. The answer is to use the endpoint that the user originally
contacted. That may just involve configuring proxies to preserve Host:
or to add a X-Forwarded-For that is used instead.

[Pablo Andres Fuente (pablo-a-fuente)]

Researching in order to understand this bug I found that there is another header that could be handy: X-Forwarded-Proto, which contains the protocol used before the proxy removed the ssh thing (in this case is 'https').
So if the webob.Request instance is modified in order to use this schema instead of the one defined by Heat API, the call to webob.Request.relative_url() will return a URL with the correct schema.
I think that the X-Forwarded-For won't be needed, because webob uses the env(HTTP_HOST) to create a URL in webob.Request.relative_url(), and that environment variable is the same that comes in X-Forwarded-For.
This modification could be done in heat.api.openstack.v1.util.make_url() or a new middleware could be added in order to alter the request. Any though about the best place to implement the change?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/64142

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/64142
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=29807237cb87e337f95cc2c46cf434b09515a9b3
Submitter: Jenkins
Branch: master

commit 29807237cb87e337f95cc2c46cf434b09515a9b3
Author: Pablo Andres Fuente <email address hidden>
Date: Fri Dec 27 10:07:04 2013 -0300

    New middleware to handle SSL termination proxies

    The Heat API doesn't behave properly if it is behind an SSL termination
    proxy. If this is the case, the HTTP redirections and the links
    returned in the REST response bodies are build using http protocol
    instead of https.

    To handle this situation, a new middleware was added. The purpose of the
    SSLMiddleware is to update the wsgi.url_scheme environment variable of
    the request with the value contained in an HTTP header that can be
    configured in heat.conf (by default: 'X-Forwarded-Proto')

    Change-Id: I418cd8cf2fb2a5fb3f630b4945084fb6eab57310
    Closes-Bug: #1235555

[Louis Watta (lwatta)]

What are the chances of this fix being back ported to Havana?

[Steve Baker (steve-stevebaker)]

This seems like a good candidate for backport

[Christopher H. Laco (claco)]

Ideally, this should be in every api across the board imho. Any service being proxied in a method other than running wsgi directly will have these issues.

In nova, you can really cheat, and use osapi_compute_link_prefix. At least with the above middleware, any localhost access bypassing the proxy will retain the proper protocol/hostname.