Cinder and glance client have endpoint selection issues

[Bjoern]

The cinder and glance client past 10.1.2 have endpoint selection issues and issues using the right protocol when using HTTPS and selecting publicURL endpoints.

The issue can be reproduced if we set OS_ENDPOINT_TYPE to publicURL and OS_AUTH_URL to the public keystone URL while using the HTTPS protocol.
The glance and cinder client seem to use the the href from the links array

DEBUG:keystoneclient.session:REQ: curl -g -i -X GET https://<ip>:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG:keystoneclient.session:RESP: [200] content-length: 424 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.7 (Ubuntu) connection: Keep-Alive date: Tue, 07 Jul 2015 14:30:23 GMT content-type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://<ip>:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://<ip>:5000/v2.0/tokens
ERROR: Unable to establish connection to http://<ip>:5000/v2.0/tokens

I have not found an upstream issue yet but I suspect that either X-FORWARDED-PROTO is needed to help the keystone middleware selecting the protocol or we need to overwrite public_endpoint inside keystone.conf

Not working :
python-cinderclient==1.1.1
python-glanceclient==0.15.0

Working :
python-glanceclient==0.13.1
python-cinderclient==1.0.9

[Peter Sabaini (peter-sabaini)]

This happens here independently of selecting https or endpoint type. However I call cinder, it ends up requesting tokens from what I suspect is the catalog entry of keystone instead of using os-auth-url

./bin/cinder --debug --os-auth-url=http://example-front.com/v2.0 list
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://example-front.com/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG:keystoneclient.session:RESP: [200] content-length: 615 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.7 (Ubuntu) connection: Keep-Alive date: Fri, 02 Oct 2015 12:55:19 GMT content-type: application/json x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://internal.ip.address:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/content/", "type": "text/html", "rel": "describedby"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/identity-dev-guide-2.0.pdf", "type": "application/pdf", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://internal.ip.address:5000/v2.0/tokens
^C... terminating cinder client

Cinder is installed via pip:

./bin/cinder --version
1.4.0

[Bjoern (bjoern-t)]

FYI, my current workaround is to configure public_endpoint inside keystone.conf to use the desired URL. That will update the href attribute inside the keystone catalog or token response.

[Major Hayden (rackerhacker)]

Is this still an issue in Kilo or Liberty?

[Bjoern (bjoern-t)]

Yes this is still an issue

[Kairat Kushaev (kkushaev)]

So glance ignores endpoint type flag IIUC. Right?
Then it may be fixed here for glanceclient: https://review.openstack.org/#/c/301214/

[Bjoern (bjoern-t)]

Yes correct. I also recall that glance was using GLANCE_ENDPOINT_TYPE, not sure if that still exist. This issue is perhaps present on more clients, possibly all of them

[Jesse Pretorius (jesse-pretorius)]

This is not a bug in OpenStack-Ansible, but in the clients. I'm therefore marking this as invalid for OSA.

[Matthew Edmonds (edmondsw)]

I think this is actually working as designed. I would view comment 2 as the fix for a misconfiguration (i.e. user error) issue, not a workaround. Per the description, the clients are calling keystone's GET https://<ip>:5000/v2.0 and yet the response they get back from keystone wrongly shows http instead of https. If they get bad information from keystone, it's harsh to blame the clients...

[Matthew Edmonds (edmondsw)]

if there is anything to fix here, it is probably in keystoneclient

[Morgan Fainberg (mdrnstm)]

this bug has had no traction, is this still the case with the KSA sessions? If so, please re0open against keystoneauth.

[Morgan Fainberg (mdrnstm)]

This would be something to fix in keystoneauth if it is still an issue. marking as invalid/wont fix.