Issue if public and internal endpoints are different

Bug #1414144 reported by Ken Thomas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Incomplete
Medium
Deliang Fan

Bug Description

We have a setup where our service public endpoints are HTTPS (to a host running Apache Traffic Server which then maps the request to use HTTP to the actual service) while the internal and admin endpoints are HTTP that should never be used off the Openstack hosts.

The problem is that make_url in util.py (https://github.com/openstack/heat/blob/master/heat/api/openstack/v1/util.py#L69) is using the current request's URL (which could be the internal URL) to generate the redirect URL to send back to the client. In our case, we end up with the client getting a URL that it can't access and <BOOM>.

We're seeing this in Icehouse and it doesn't looks like the code has changed in Juno.

We're currently doing a local patch to fix this (below is the exact code that we're currently using for make_url). We'd like to...

1. Fix this in a less hacky way if possible.
2. Get the fix into the upstream code base in case other folks have different URLs for public and internal endpoints *and* allow us to get rid of a local patch.

I'm happy to work on the fix for this. Just let me know how you'd would prefer to go about it.

Thanks,
Ken

def make_url(req, identity):
    '''Return the URL for the supplied identity dictionary.'''
    try:
        stack_identity = identifier.HeatIdentifier(**identity)
    except ValueError:
        err_reason = _('Invalid Stack address')
        raise exc.HTTPInternalServerError(err_reason)
    ########### BEGIN PATCH
    # See if we can get the public endpoint from the service catalog
    # in the headers. Yes, we could try to put it into the request context
    # but we're going to process it per request, and if we put it there
    # (1) We'd process it for every request and not just the ones that
    # need it.
    # (2) It would complicate this patch by spreading it across multiple
    # files
    # If we can't get the service catalog or find the public endpoint for
    # heat, then use the old logic.
    try:
        sc = json_loads(req.headers['X-Service-Catalog'])
        if sc:
            # Walk through the entries and look for the 'heat' info
            for entry in sc:
                if entry.get('name') == 'heat':
                    # The endpoints will be an array of dicts, walk through em
                    for endpoint_data in entry.get('endpoints'):
                        pu = endpoint_data.get('publicURL')
                        if pu:
                            # If we found the public URL, let urlparse
                            # do the work of splicing it to the stack info
                            newurl = urlparse.urljoin(
                                pu,
                                stack_identity.url_path())
                            logger.debug('Used public endpoint::%s' % newurl)
                            return newurl
    except Exception, ex:
        # This is okay. It just means that we had a problem getting the
        # service catalog. We'll just ignore it and use the pre-patch code
        logger.debug("No service catalog, do it the old way, ex:%s" % ex)
    ########### END PATCH
    return req.relative_url(stack_identity.url_path(), True)

Angus Salkeld (asalkeld)
Changed in heat:
importance: Undecided → Medium
status: New → Triaged
Changed in heat:
status: Triaged → Confirmed
assignee: nobody → Deliang Fan (vanderliang)
Revision history for this message
Marcus Furlong (furlongm) wrote :
Download full text (3.7 KiB)

We have a similar set up.

Would this bug be the reason we are seeing the following in horizon?

2015-04-01 08:16:39,837 8800 ERROR django.request Internal Server Error: /dashboard/project/stacks/
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 109, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 86, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/django/views/generic/base.py", line 48, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/django/views/generic/base.py", line 69, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 162, in post
    return self.get(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 121, in handle_table
    handled = self._tables[name].maybe_handle()
  File "/usr/lib/python2.6/site-packages/horizon/tables/base.py", line 1480, in maybe_handle
    return self.take_action(action_name, obj_id)
  File "/usr/lib/python2.6/site-packages/horizon/tables/base.py", line 1326, in take_action
    response = action.multiple(self, self.request, obj_ids)
  File "/usr/lib/python2.6/site-packages/horizon/tables/actions.py", line 294, in multiple
    return self.handle(data_table, request, object_ids)
  File "/usr/lib/python2.6/site-packages/horizon/tables/actions.py", line 665, in handle
    exceptions.handle(request, ignore=ignore)
  File "/usr/lib/python2.6/site-packages/horizon/tables/actions.py", line 649, in handle
    self.action(request, datum_id)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/stacks/tables.py", line 57, in action
    api.heat.stack_delete(request, stack_id)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/heat.py", line 60, in stack_delete
    return heatclient(request).stacks.delete(stack_id)
  File "/usr/lib/python2.6/site-packages/heatclient/v1/stacks.py", line 131, in delete
    self._delete("/stacks/%s" % stack_id)
  File "/usr/lib/python2.6/site-packages/heatclient/openstack/common/apiclient/base.py", line 202, in _delete
    return self.client.delete(url)
  File "/usr/lib/python2.6/site-packages/heatclient/common/http.py", line 289, in delete
    return self.raw_request("DELETE", url, ...

Read more...

Changed in heat:
status: Confirmed → In Progress
Revision history for this message
Deliang Fan (vanderliang) wrote :
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

Can you confirm your proxy is correctly setting X-Forwarded-Host and X-Forwarded-Proto headers?

See this bug for a similar issue which was fixed by changing haproxy configuration:
https://bugzilla.redhat.com/show_bug.cgi?id=1201227#c14

Changed in heat:
status: In Progress → Incomplete
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Deliang Fan (<email address hidden>) on branch: master
Review: https://review.openstack.org/174775
Reason: Heat has already fix this problem if we configure haproxy properly.

Revision history for this message
Ken Thomas (krt) wrote :

I'll check our set up when I can. (It will be a while as I've been pulled into another project...) We're using Apache Traffic Server rather than haproxy, but it sounds like if headers are *NOT* there, it's either an issue with ATS or the way we configured it.

Thanks all!

Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.