Fuel for OpenStack

[heat][ssl] User can't get info about stack via horizon

Bug #1486611 reported by Anastasia Kuznetsova on 2015-08-19

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Released	High	Denis Egorenko	Fuel for OpenStack 7.0
6.0.x	Won't Fix	High	Sergii Rizvan	Fuel for OpenStack 6.0-updates
6.1.x	Invalid	High	Sergii Rizvan	Fuel for OpenStack 6.1-updates

Bug Description

Steps to reproduce:
1. Create some stack via horizon or cli
2. After its creation try to get info about it by clicking on its name in horizon

Actual result:
nvalidEndpoint at /project/stacks/stack/ded137fe-17e9-49ff-846c-91cbeaf6b8cb/
Prohibited endpoint redirect http://public.fuel.local:8004/v1/810bc14559e443219f407133496461a3/stacks/some-stack/ded137fe-17e9-49ff-846c-91cbeaf6b8cb
Request Method: GET
Request URL: https://172.18.78.48:8077/horizon/project/stacks/stack/ded137fe-17e9-49ff-846c-91cbeaf6b8cb/
Django Version: 1.6.1
Exception Type: InvalidEndpoint
Exception Value:
Prohibited endpoint redirect http://public.fuel.local:8004/v1/810bc14559e443219f407133496461a3/stacks/some-stack/ded137fe-17e9-49ff-846c-91cbeaf6b8cb
Exception Location: /usr/lib/python2.7/dist-packages/heatclient/common/http.py in strip_endpoint, line 241

ISO:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "187"
  build_id: "2015-08-18_03-05-20"
  nailgun_sha: "4710801a2f4a6d61d652f8f1e64215d9dde37d2e"
  python-fuelclient_sha: "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e24ca066bf6160bc1e419aaa5d486cad1aaa937d"
  fuel-library_sha: "0062e69db17f8a63f85996039bdefa87aea498e1"
  fuel-ostf_sha: "17786b86b78e5b66d2b1c15500186648df10c63d"
  fuelmain_sha: "c9dad194e82a60bf33060eae635fff867116a9ce"

ENV: 3 controllers, 1 compute, TLS fully enabled

Tags:

Anastasia Kuznetsova (akuznetsova) on 2015-08-19

Changed in fuel:
importance:	Undecided → High
assignee:	nobody → MOS Heat (mos-heat)
milestone:	none → 7.0

Revision history for this message

Anastasia Kuznetsova (akuznetsova) wrote on 2015-08-19:

I've tried to fix it by adding:
http-request set-header X-Forwarded-Proto https if { ssl_fc }
to /etc/haproxy/conf.d/160-heat-api.cfg and futher restart of haproxy. But it didn't help or maybe I missed something else

Sergey Kraynev (skraynev) on 2015-08-19

Changed in fuel:
assignee:	MOS Heat (mos-heat) → Peter Razumovsky (prazumovsky)

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-20:

Download full text (5.6 KiB)

The issue is reproducible via CLI too:
heat --debug --os-username admin --os-password admin --os-tenant-name admin --os-auth-url https://public.fuel.local:5000/v2.0/ --os-cacert cert stack-show 1c914fed-0264-47de-8a94-f4be5ebc4790

DEBUG (session) REQ: curl -g -i --cacert "/home/ishakhat/Work/cert" -X GET https://public.fuel.local:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTPS connection (1): public.fuel.local
/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
DEBUG (connectionpool) "GET /v2.0/ HTTP/1.1" 200 344
DEBUG (session) RESP: [200] content-length: 344 vary: X-Auth-Token server: Apache connection: close date: Wed, 19 Aug 2015 15:56:45 GMT content-type: application/json x-openstack-request-id: req-8949891f-2aca-4a49-8216-ca2c600402e4
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://public.fuel.local:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to https://public.fuel.local:5000/v2.0/tokens
INFO (connectionpool) Resetting dropped connection: public.fuel.local
/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 4452
DEBUG (session) REQ: curl -g -i --cacert "/home/ishakhat/Work/cert" -X GET https://public.fuel.local:8004/v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 -H "User-Agent: python-heatclient" -H "Content-Type: application/json" -H "X-Auth-Url: https://public.fuel.local:5000/v2.0/" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}28ec32ed16cecfd79640b503b3366c08ee579d79"
INFO (connectionpool) Starting new HTTPS connection (1): public.fuel.local
/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SecurityWarning
DEBUG (connectionpool) "GET /v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 HTTP/1.1" 302 233
DEBUG (session) RESP: [302] content-length: 233 connection: close location: http://public.fuel.local:8004/v1/4b765fba209...

DEBUG (session) REQ: curl -g -i --cacert "/home/ishakhat/Work/cert" -X GET https://public.fuel.local:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTPS connection (1): public.fuel.local
/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
DEBUG (connectionpool) "GET /v2.0/ HTTP/1.1" 200 344
DEBUG (session) RESP: [200] content-length: 344 vary: X-Auth-Token server: Apache connection: close date: Wed, 19 Aug 2015 15:56:45 GMT content-type: application/json x-openstack-request-id: req-8949891f-2aca-4a49-8216-ca2c600402e4 
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://public.fuel.local:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to https://public.fuel.local:5000/v2.0/tokens
INFO (connectionpool) Resetting dropped connection: public.fuel.local
/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 4452
DEBUG (session) REQ: curl -g -i --cacert "/home/ishakhat/Work/cert" -X GET https://public.fuel.local:8004/v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 -H "User-Agent: python-heatclient" -H "Content-Type: application/json" -H "X-Auth-Url: https://public.fuel.local:5000/v2.0/" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}28ec32ed16cecfd79640b503b3366c08ee579d79"
INFO (connectionpool) Starting new HTTPS connection (1): public.fuel.local
/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
DEBUG (connectionpool) "GET /v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 HTTP/1.1" 302 233
DEBUG (session) RESP: [302] content-length: 233 connection: close location: http://public.fuel.local:8004/v1/4b765fba209b453d9c755b1f2bcb7664/stacks/shaker_b1e4d550-c67e-4ec2-8a7d-e416870732d6/1c914fed-0264-47de-8a94-f4be5ebc4790 date: Wed, 19 Aug 2015 15:56:46 GMT content-type: text/plain; charset=UTF-8 x-openstack-request-id: req-a74f76b4-c08f-4c2e-af84-5dcf6683eec4 
RESP BODY: 302 Found

The resource was found at http://public.fuel.local:8004/v1/4b765fba209b453d9c755b1f2bcb7664/stacks/shaker_b1e4d550-c67e-4ec2-8a7d-e416870732d6/1c914fed-0264-47de-8a94-f4be5ebc4790; you should be redirected automatically.

INFO (connectionpool) Starting new HTTP connection (1): public.fuel.local
Traceback (most recent call last):
  File ".venv/bin/heat", line 11, in <module>
    sys.exit(main())
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/heatclient/shell.py", line 705, in main
    HeatShell().main(args)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/heatclient/shell.py", line 655, in main
    args.func(client, args)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/heatclient/v1/shell.py", line 375, in do_stack_show
    stack = hc.stacks.get(**fields)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 202, in get
    resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/heatclient/common/http.py", line 265, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/heatclient/common/http.py", line 360, in _http_request
    raise_exc=False, **kwargs)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 382, in request
    resp = send(**kwargs)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 466, in _send_request
    **kwargs)
  File "/home/ishakhat/Work/shaker/.venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 426, in _send_request
    raise exceptions.ConnectionRefused(msg)
keystoneclient.openstack.common.apiclient.exceptions.ConnectionRefused: Unable to establish connection to http://public.fuel.local:8004/v1/4b765fba209b453d9c755b1f2bcb7664/stacks/shaker_b1e4d550-c67e-4ec2-8a7d-e416870732d6/1c914fed-0264-47de-8a94-f4be5ebc4790

ruhe (ruhe) on 2015-08-20

tags:

added: heat

Revision history for this message

Sergey Kraynev (skraynev) wrote on 2015-08-20:

Heat team tries to debug the issue

Changed in fuel:
status:	New → Triaged

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-20:

Tried to update haproxy config with:
http-request add-header X-Forwarded-Proto https if { ssl_fc }
With this option some portion (~20%) of requests succeed, but other fail. Traffic capture collected between haproxy and heat-api service shows that the header is set not for all requests, e.g. it's observed the following:

request #1
--------------
GET /v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 HTTP/1.1
Host: public.fuel.local:8004
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-heatclient
X-Auth-Token: 90c4a57ae23c416ea86b5a08bfeab400
Content-Type: application/json
X-Auth-Url: https://public.fuel.local:5000/v2.0/
Connection: close

request #2
--------------
GET /v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 HTTP/1.1
Host: public.fuel.local:8004
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-heatclient
X-Auth-Token: 0e4ee4c13d88479f8d2ae52af8dcb33e
Content-Type: application/json
X-Auth-Url: https://public.fuel.local:5000/v2.0/
X-Proto: https
Connection: close

request #3
---------------
GET /v1/4b765fba209b453d9c755b1f2bcb7664/stacks/1c914fed-0264-47de-8a94-f4be5ebc4790 HTTP/1.1
Host: public.fuel.local:8004
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-heatclient
X-Auth-Token: 6530da1fee8f4db281e5edd9014c6ae9
Content-Type: application/json
X-Auth-Url: https://public.fuel.local:5000/v2.0/
X-Forwarded-Proto: https
Connection: close

From the above it looks like the issue is in haproxy

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-20:

The issue observed in https://bugs.launchpad.net/fuel/+bug/1486611/comments/4 is caused by https://bugs.launchpad.net/fuel/+bug/1487056. Thus the fix proposed in https://bugs.launchpad.net/fuel/+bug/1486611/comments/1 works

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-20: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/215224

Changed in fuel:
assignee:	Peter Razumovsky (prazumovsky) → Denis Egorenko (degorenko)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-21: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/215224
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7ea979f23d1b29bb9998ff57058d463dc3ca521e
Submitter: Jenkins
Branch: master

commit 7ea979f23d1b29bb9998ff57058d463dc3ca521e
Author: Denis Egorenko <email address hidden>
Date: Thu Aug 20 19:15:44 2015 +0300

Change request header in HaProxy configuration for Heat API

    Set requests header for Heat API service in HaProxy configuration to
    'https' if ssl is used. It will help to avoid the problem with
    wrong redirect during getting some info from Heat.

Change-Id: Ia2ce1d9965d571768f8ec26763b3d3afe4e19bdd
Closes-bug: #1486611

Changed in fuel:
status:	In Progress → Fix Committed

Alexander Bochkarev (abochkarev) on 2015-09-03

tags:

added: on-verification

Revision history for this message

Alexander Bochkarev (abochkarev) wrote on 2015-09-03:

Verified with fuel-7.0-257-2015-08-30_17-23-00.iso

tags:	removed: on-verification
Changed in fuel:
status:	Fix Committed → Fix Released

Revision history for this message

Roman Rufanov (rrufanov) wrote on 2015-09-17:

customer found on 6.0

tags:

added: customer-found support

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-10-21:

#10

Won't Fix for 6.0-updates because puppets are not packaged in 6.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-22: Fix proposed to fuel-library (stable/6.1)

#11

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/238435

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-22:

#12

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/238447

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-29: Change abandoned on fuel-library (stable/6.1)

#13

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/238435
Reason: Abandoned in favor of: https://review.openstack.org/238447

Revision history for this message

Sergii Rizvan (srizvan) wrote on 2016-01-19:

#14

Closing as invalid for 6.1 because we don't have SSL in 6.1.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-01-19:

#15

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/238447

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.