Keystone forwards to public URL despite OS_ENDPOINT_TYPE value

[Ilya Shakhat]

openrc:
export LC_ALL=C
export OS_NO_CACHE='true'
export OS_TENANT_NAME='admin'
export OS_PROJECT_NAME='admin'
export OS_USERNAME='admin'
export OS_PASSWORD='admin'
export OS_AUTH_URL='http://192.168.0.2:5000/v2.0/'
export OS_DEFAULT_DOMAIN='default'
export OS_AUTH_STRATEGY='keystone'
export OS_REGION_NAME='RegionOne'
export CINDER_ENDPOINT_TYPE='internalURL'
export GLANCE_ENDPOINT_TYPE='internalURL'
export KEYSTONE_ENDPOINT_TYPE='internalURL'
export NOVA_ENDPOINT_TYPE='internalURL'
export NEUTRON_ENDPOINT_TYPE='internalURL'
export OS_ENDPOINT_TYPE='internalURL'

$ neutron --debug port-list
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://192.168.0.2:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG: keystoneclient.session RESP: [200] Content-Length: 344 Vary: X-Auth-Token Server: Apache Connection: close Date: Wed, 21 Oct 2015 14:22:33 GMT Content-Type: application/json x-openstack-request-id: req-79c0eedf-400a-4093-8917-a9b6ddd305ba
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://public.fuel.local:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

It's expected that Keystone returns internal endpoint, but instead it forwards to the public (https://public.fuel.local:5000/v2.0/)

The issue appears in MOS 8.0 and 7.0, but not in 6.1

[Ilya Shakhat (shakhat)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"
  nailgun_sha: "4162b0c15adb425b37608c787944d1983f543aa8"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"

[Ilya Shakhat (shakhat)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "50"
  build_id: "50"
  fuel-nailgun_sha: "491bff70fb5a05e97b9d1daaa706aa5d16cefef7"
  python-fuelclient_sha: "952c9d9ba5ba31213328cc450dec54395cb580e1"
  fuel-agent_sha: "49e2e8a08a3cf39d1b38ee155015332aa84e47fd"
  fuel-nailgun-agent_sha: "abab45cf8c7344d43acd3858c02d7a648ef7fee6"
  astute_sha: "7cea69bd021d6132cde473cdd25e5142fdddad95"
  fuel-library_sha: "1b5667d4093a7406367c02f94e8b84f40048ffa6"
  fuel-ostf_sha: "618eb749d78cce075c2b14664b59a06d92f142c2"
  fuel-createmirror_sha: "843736e50534d184f4b07744d68d0f2d27284cae"
  fuelmain_sha: "eb669555ca18b0fb372c53857f1c7b2f10e669dd"

[Boris Bobrov (bbobrov)]

Have you seen https://bugs.launchpad.net/python-neutronclient/+bug/1368676 ? Can you confirm that the patch that fixed the bug is included in our distro? What makes you think that it's a bug in keystone(client) rather then in neutronclient?

[Ilya Shakhat (shakhat)]

Boris, yes, the patch is present. It was included into 2.3.10 and MOS 7.0 contains version 2.4.0

The same issue is for nova client (and actually for all that based on Keystone Session):
$ nova --debug image-list
DEBUG (session:195) REQ: curl -g -i -X GET http://192.168.0.4:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:187) Starting new HTTP connection (1): 192.168.0.4
DEBUG (connectionpool:353) "GET /v2.0/ HTTP/1.1" 200 344
DEBUG (session:224) RESP: [200] content-length: 344 vary: X-Auth-Token server: Apache connection: close date: Thu, 22 Oct 2015 13:06:54 GMT content-type: application/json x-openstack-request-id: req-8ab587b7-4423-4922-8295-eed1cb730607
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://public.fuel.local:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
DEBUG (v2:76) Making authentication request to https://public.fuel.local:5000/v2.0/tokens
INFO (connectionpool:663) Starting new HTTPS connection (1): public.fuel.local

If you suspect that the issue is in the way how clients use python-keystoneclient, please provide guidelines on correct usage of it. Assigning the bug back to mos-keystone team.

[Davanum Srinivas (DIMS) (dims-v)]

Boris, Ilya, yes, this is odd, we should dig into it.

[Davanum Srinivas (DIMS) (dims-v)]

fyi, i don't see this behavior in latest devstack

[Matthew Mosesohn (raytrac3r)]

This is not a client behavior but a keystone server behavior. It's returning this public address regardless of which endpoint accesses the API. We ran into https://bugs.launchpad.net/fuel/+bug/1517054 today, which is affecting deployments where keystone isn't on the controller. It's important that this works correctly, so that public network isn't required for deployment.

[Alexander Makarov (amakarov)]

Keystone gets hostname to construct own endpoint from its config:

[DEFAULT]
public_endpoint

please check this setting

[Matthew Mosesohn (raytrac3r)]

public_endpoint configuration should be used for connections on the public address. If we don't use this setting, we'll run into a different problem.
Example with openstack --debug token issue
1 - client connects to public endpoint with HTTPS
2 - haproxy load balancer de-encapsulates SSL and forwards to keystone
3 - new URL sent back to client will be without SSL
4 - connection failed because non SSL connection is disallowed

command output with traceback: http://paste.openstack.org/show/qOT8wESwmX0G19pfZ1bB/

[Alexander Makarov (amakarov)]

What about removing that parameter at all? Keystone will use the hostname from the request.

[Alexander Makarov (amakarov)]

TBH, it's an expected behavior to return explicitly specified public endpoint to any request no matter where it came from

[Alexander Makarov (amakarov)]

If we need to change behavior, it should be a feature request with spec and so on.

[Vladimir Kuklin (vkuklin)]

Folks

Could you please point to the documentation where this behaviour was deprecated. Otherwise it is a regression, not a feature.

[Matthew Mosesohn (raytrac3r)]

This is indeed a regression from 7.0 to 8.0:
7.0 puppet output for token creation: http://paste.openstack.org/show/scAn48PgwjIFqefERraC/
8.0 puppet output for token creation: http://paste.openstack.org/show/SdeNSU4nlJBUbMslKulI/

[Alexander Makarov (amakarov)]

I've tried the same request on 7.0
curl http://192.168.0.2:5000/v3/
the result is "links": [{"href": "http://172.16.56.36:5000/v3/"
public_endpoint = http://172.16.56.36:5000
so nothing changed

I don't understand what's happening in the bug
what API is being called?

I looked in the code - nothing changed there too.

The expectation in the bug description is invalid for both 7.0 and 8.0

[Alexander Makarov (amakarov)]

Vova, what's the behavior you are about?
Nothing even changed in this call:

MOS7.0
root@node-3:~# curl http://localhost:5000/v3/
{"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "https://public.fuel.local:5000/v3/", "rel": "self"}]}}

MOS 8.0
root@node-4:~# curl http://localhost:5000/v3/
{"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "https://public.fuel.local:5000/v3/", "rel": "self"}]}}

The main mistake is to presume that keystone will return link to http://localhost:5000/v3/ in this case rather than https://public.fuel.local:5000/v3/ that is specified in the config

The root cause is not a regression in keystone but different usage of it

[Artem Hrechanychenko (agrechanichenko)]

HI!

I have failed CI job - https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.plugins.thread_2_separate_services/48/console

Steps to reproduce:
Deploy cluster with 3 nodes with db, keystone, rabbit, horizon

Scenario:
1. Create cluster
2. Add 3 nodes with controller role
3. Add 3 nodes with database, keystone, rabbit,
horizon
4. Add 1 compute and cinder
5. Verify networks
6. Deploy the cluster
7. Verify networks
8. Run OSTF

Actual result - Deployment has failed. Check these nodes:
'slave-04_standalone-database_standalone-rabbitmq_standalone-keystone', 'slave-06_standalone-database_standalone-keystone_standalone-rabbitmq'

From puppet logs on failed nodes -

(/Stage[main]/Keystone::Roles::Admin/Keystone_user[separateall]) Could not evaluate: Execution of '/usr/bin/openstack token issue --format value' returned 1: ERROR: openstack Unable to establish connection to https://10.109.1.3:5000/v3/auth/tokens

[Ilya Shakhat (shakhat)]

This issue is about Keystone forwarding to public endpoint when being accessed through internal.

openrc file:
export OS_AUTH_URL='http://192.168.0.2:5000/v2.0/'
export OS_AUTH_STRATEGY='keystone'
export KEYSTONE_ENDPOINT_TYPE='internalURL'

$ curl http://192.168.0.2:5000/v2.0/
{"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://172.16.46.56:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

I'd expect Keystone to return link to http://192.168.0.2:5000/v2.0/. May the root cause be in Apache config / HTTP header processing?

Re-assigning bug back to mos-keystone to proceed with the investigation.

[Alexander Makarov (amakarov)]

Ilya, can you please point to a document describing that you may expect it?
Please read my #15 and #16 comments

Here is an excerpt from the keystone conig:

# The base public endpoint URL for Keystone that is advertised to clients
# (NOTE: this does NOT affect how Keystone listens for connections). Defaults
# to the base host URL of the request. E.g. a request to
# http://server:5000/v3/users will default to http://server:5000. You should
# only need to set this value if the base URL contains a path (e.g. /prefix/v3)
# or the endpoint should be found on a different server. (string value)
#public_endpoint = <None>

I think it describes exact that behavior we face.

[Boris Bobrov (bbobrov)]

the concern is that when we do

DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://192.168.0.7:35357/v3 -H "Accept: application/json" -H "User-Agent: python-openstackclient"
INFO: requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 192.168.0.7
DEBUG: requests.packages.urllib3.connectionpool "GET /v3 HTTP/1.1" 200 257

we get the following response:

DEBUG: keystoneclient.session RESP: [200] content-length: 257 vary: X-Auth-Token server: Apache connection: close date: Fri, 20 Nov 2015 10:50:54 GMT content-type: application/json x-openstack-request-id: req-9caf81ec-ae0b-4d2a-91df-495f1298de85
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "https://public.fuel.local:5000/v3/", "rel": "self"}]}}

and when the client looks at the link with rel "self", it makes a request to public.fuel.local:

DEBUG: keystoneclient.auth.identity.v3.base Making authentication request to https://public.fuel.local:5000/v3/auth/tokens

which is not available at the moment.

The issue is not in 7.0 because version discovery is not used there. The issue is not in keystone, it is in the clients. I am investigating it.

[Boris Bobrov (bbobrov)]

I've found https://bugs.launchpad.net/python-openstackclient/+bug/1410364 and it seems to be the same bug we're having.

[Alexander Makarov (amakarov)]

Boris, we should keep looking: the bug is valid when public_endpoint is localhost, - it's not our case.

[Sergii Golovatiuk (sgolovatiuk)]

In case of HAProxy all headers should be passed to the server to see the real IP instead of IP of HAProxy. However, we tested the behavior with haproxy it remains the same internal > public > internal

[Boris Bobrov (bbobrov)]

As far as I understood, commenting out public_endpoint helped. Was this done in some patch?

[Alexander Makarov (amakarov)]

Ok, the situation:

* client app using keystoneclient library tries to do something
* for that it needs a token
* for that it needs a keystone endpoint to request
* now it requests the endpoint from keystone based on its OS_AUTH_URL env var
* keystone returns public_endpoint specified in its config
* client app then tries to request a token using this endpoint
* the endpoint is inaccessible

This can be solved several ways depending on the needs.

So the question is: what is the needed behavior? I can say that ks and ksc work as designed, so if we want to change something, it will not be a bug fix.

[Matthew Mosesohn (raytrac3r)]

I am blocked by this bug because we have a situation where we are deploying a keystone standalone node. It can't do public SSL because there is exactly 1 host that does all public traffic termination (controller) - and this node must be deployed before controller nodes. This is an apparent regression in behavior where openstack client was able to obtain a token using internal URL only.

At the same time, keystone must upgrade public_endpoint traffic to SSL. If you remove this option, Keystone connections fail because it responds with non-SSL connections. There needs to be a way to do SSL upgrading for publicURL connections, but not block internalURL connections if public endpoint isn't accessible

[Matthew Mosesohn (raytrac3r)]

Secondly, please kindly don't move this bug back into Fuel-Library unless there's a clear task we can perform on our side. I will receive notifications without needing to reassign every iteration.

[Matthew Mosesohn (raytrac3r)]

Sergii and Boris discussed disabling the public_endpoint URL parameter in keystone. That causes keystone to return non-SSL responses to API requests (like openstack token issue), then the connection will fail because you must use SSL to connect:
http://paste.openstack.org/show/rWzE9cMTrYiSFzmXNTa9/

Disabling public_endpoint breaks public SSL, so it's not a complete fix.

[Alexander Makarov (amakarov)]

Can you enable it later in the deployment process?

[Boris Bobrov (bbobrov)]

From my talk with Matthew

OK, the situation then.
There is no SSL on internal APIs, there is SSL on public APIs. Deployment happens via internal APIs. Public APIs are not available at the moment of deployment.
If public_endpoint is commented and request comes from internal network, some magic happens and correct addresses are returned.
If public_endpoint is commented and request comes from public network, the magic works wrong and URL returned from keystone is non-https and nothing works after that.
If public_endpoint is set to public.fuel.local and request comes from internal network, then public.fuel.local is not available at that moment and nothing works.

How things work on the side of keystoneclient:
1. It queries OS_AUTH_URL (keystone) for version discovery. OS_AUTH_URL (keystone) returns a response with an url, which is public_endpoint if it is set or host_url if it is not set. There is a comment about that in keystone code:

# NOTE(jamielennox): if url is not set via the config file we
# should set it relative to the url that the user used to get here
# so as not to mess with version discovery. This is not perfect.
# host_url omits the path prefix, but there isn't another good
# solution that will work for all urls.
url = context['host_url']

2. The client passes this url to an auth plugin.

3. The auth plugin appends something (/auth/token) to the url and queries the new url

Some folks had a similar problem. They had `public_endpoint = http://localhost` and were bootstraping keystone from the outside. There is a bugreport and a fix for that: https://bugs.launchpad.net/python-openstackclient/+bug/1410364 . The fix doesn't work for us.

We cannot cleanly change public_endpoint after the deployment when public network is ready

I see several options:

1. Change the proposed fix to work not only for localhost, but for everything else. You'll need to specify a plugin for `openstack` that will be used during deployment and switch it back after the deployment. I don't know how to specify a plugin though.

2. Enable public endpoint during deployment. I have no idea how hard it would be

3. Do something on keystone side. Open a bugreport there, spec (because it will change API), fix, wait for a couple of months until it is accepted.

[Boris Bobrov (bbobrov)]

in fact, setting public_endpoint basically kills the idea of public/internal APIs, because it will always be either the first one or the second. To have a separation to public/internal APIs there should be a way to pass the requested URL from haproxy to keystone.

[Matthew Mosesohn (raytrac3r)]

After some research, we've decided that public_endpoint really breaks the ability to separate internal and public networks. We found the option secure_proxy_ssl_header from https://review.openstack.org/#/c/132235/18 that allows keystone to return the proper protocol when placed behind an SSL terminator proxy.

The steps to fix are: add a header in haproxy if SSL is enabled, add this secure_proxy_ssl_header to keystone.conf, and remove public_endpoint setting from keystone.conf (mark as unset)

[Bogdan Dobrelya (bogdando)]

Revert submitted https://review.openstack.org/#/c/250806/

[Matthew Mosesohn (raytrac3r)]

This patch didn't introduce the regression: https://review.openstack.org/#/c/249693/ did

[Boris Bobrov (bbobrov)]

Matthew, it seems that the issue is still there: https://bugs.launchpad.net/mos/+bug/1526881

[Boris Bobrov (bbobrov)]

Everyone: Please see https://bugs.launchpad.net/keystone/+bug/1370022 for a workaround

[Matthew Mosesohn (raytrac3r)]

@Boris this was already committed in https://review.openstack.org/#/c/248823/ and was thoroughly tested. The code committed in keystone bug 1370022 is already applicable and in keystone in Liberty.

 I'm not sure if this is really a duplicate bug of https://bugs.launchpad.net/keystone/+bug/1370022 . I need more info from the reporter.

[Matthew Mosesohn (raytrac3r)]

For now moving this back to fix committed and 1370022 to incomplete and assigning to myself.

[Veronica Krayneva (vkrayneva)]

Verified on
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "362"
  build_id: "362"
  fuel-nailgun_sha: "53c72a9600158bea873eec2af1322a716e079ea0"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "7463551bc74841d1049869aaee777634fb0e5149"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "ba8063d34ff6419bddf2a82b1de1f37108d96082"
  fuel-ostf_sha: "889ddb0f1a4fa5f839fd4ea0c0017a3c181aa0c1"
  fuel-mirror_sha: "8adb10618bb72bb36bb018386d329b494b036573"
  fuelmenu_sha: "824f6d3ebdc10daf2f7195c82a8ca66da5abee99"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "07d5f1c3e1b352cb713852a3a96022ddb8fe2676