Keystone API GET 5000/v3 returns wrong endpoint URL in response body

[Yang Zhang]

When I was invoking a GET request to public endpoint of Keystone, I found the admin endpoint URL in response body, I assume it should be the public endpoint URL:
GET https://192.168.101.10:5000/v3

{
  "version": {
    "status": "stable",
    "updated": "2013-03-06T00:00:00Z",
    "media-types": [
      {
        "base": "application/json",
        "type": "application/vnd.openstack.identity-v3+json"
      },
      {
        "base": "application/xml",
        "type": "application/vnd.openstack.identity-v3+xml"
      }
    ],
    "id": "v3.0",
    "links": [
      {
        "href": "https://172.20.14.10:35357/v3/",
        "rel": "self"
      }
    ]
  }
}

===============================================================
Btw, I can get the right URL for public endpoint in the response body of the versionless API call:
GET https://192.168.101.10:5000

{
  "versions": {
    "values": [
      {
        "status": "stable",
        "updated": "2013-03-06T00:00:00Z",
        "media-types": [
          {
            "base": "application/json",
            "type": "application/vnd.openstack.identity-v3+json"
          },
          {
            "base": "application/xml",
            "type": "application/vnd.openstack.identity-v3+xml"
          }
        ],
        "id": "v3.0",
        "links": [
          {
            "href": "https://192.168.101.10:5000/v3/",
            "rel": "self"
          }
        ]
      },
      {
        "status": "stable",
        "updated": "2014-04-17T00:00:00Z",
        "media-types": [
          {
            "base": "application/json",
            "type": "application/vnd.openstack.identity-v2.0+json"
          },
          {
            "base": "application/xml",
            "type": "application/vnd.openstack.identity-v2.0+xml"
          }
        ],
        "id": "v2.0",
        "links": [
          {
            "href": "https://192.168.101.10:5000/v2.0/",
            "rel": "self"
          },
          {
            "href": "http://docs.openstack.org/api/openstack-identity-service/2.0/content/",
            "type": "text/html",
            "rel": "describedby"
          },
          {
            "href": "http://docs.openstack.org/api/openstack-identity-service/2.0/identity-dev-guide-2.0.pdf",
            "type": "application/pdf",
            "rel": "describedby"
          }
        ]
      }
    ]
  }
}

[Yang Zhang (bjzyang)]

This was found in IceHouse Keystone-2014.1.2

[Adam Young (ayoung)]

The problem is in keystone/service.py and pulling in keystone/controller.py.

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/controllers.py#n117

self.base_url comes from wsgi:

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/wsgi.py#n355

 admin_url or main_url needs to be set correctly in the config file. Otherwise, it falls back to host_url.

[Dolph Mathews (dolph)]

The :5000/v3 and :35357/v3 are both identical in terms of functionality. It doesn't really matter which one is exposed, but it should be consistent with the requested interface (so, the port should match the request).

[Yang Zhang (bjzyang)]

Yes, and the IP in self link will also be inconsistent if we are using different IP for public and admin URLs of endpoint. That will cause more problem as described below:

The main effect of this is that some standard tooling and possibly certain automation frameworks will break, as they use the self link for further calls.

For example, python-openstackclient supports identity v3 API, and it's keystone management using v3 API commands will fail to execute because after initial token it tries to connect to the wrong network which is for admin URL.
Example:

$ openstack user list
INFO: urllib3.connectionpool Starting new HTTPS connection (1): 192.168.101.10
INFO: urllib3.connectionpool Starting new HTTPS connection (1): 172.20.14.10
(times out after that)

(192.168.101.10 above is for public URL, 172.20.14.10 is for admin URL)

So I would suggest to raise the importance of this bug. Thanks!

[Alexey Miroshkin (amirosh)]

Yang, do you have public_endpoint and admin_endpoint explicitly set? So far I was able to reproduce this bug only when these parameters are set.

[Yang Zhang (bjzyang)]

Alexey, exactly, we have public_endpoint and admin_endpoint explicitly set in keystone.conf.
(That is required in our deployment to avoid issue of python clients since we are using different network for public_endpoint and admin_endpoint).

[Alexey Miroshkin (amirosh)]

Mu understanding of Keystone is rudimentary, please comment to clarify it for me.

If public_endpoint or admin_endpoint is not set as Adam mentioned host_url is used and we get the correct url (in case if there is no addition in path, but it's a different story). But in fact this situation hides a real problem.

If these values are set, endpoint_type is used to construct the url and endpoint_type is *always* 'admin' now. In service.py

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/service.py#n114

we create an app with 2 identical routers inside and from my experience the first one is always returned to process a request. I've changed order of append() and started get 'public' Version app always. Obviously, I'm missing something here, but I don't understand that design.

We always deploy 2 (sub)apps (admin and public) on every server, even if admin_bind_host and public_bind_host (deprecated, what should I use instead?) are specified. Those 2 apps identical and the only one is used. So, perhaps we should get rid of the second one?

[Yang Zhang (bjzyang)]

Btw, I also found the similar issue in Nova (but other components seem to be fine):

GET https://192.168.101.10:8774

{
  "versions": [
    {
      "status": "CURRENT",
      "updated": "2011-01-21T11:33:21Z",
      "id": "v2.0",
      "links": [
        {
          "href": "https://172.20.14.10:8774/v2/",
          "rel": "self"
        }
      ]
    },
    {
      "status": "EXPERIMENTAL",
      "updated": "2013-07-23T11:33:21Z",
      "id": "v3.0",
      "links": [
        {
          "href": "https://172.20.14.10:8774/v3/",
          "rel": "self"
        }
      ]
    }
  ]
}

[Yang Zhang (bjzyang)]

Hi Alexey, how is the status of the bug fix?

[Yang Zhang (bjzyang)]

The expected result is that the IP in response body should be the same with the IP in the request URL.

[Ciaran O Tuathail (ciaran-otuathail)]

It looks like this has been addressed in Commit https://review.openstack.org/#/c/118522/

[Alexey Miroshkin (amirosh)]

Hi Yang, fix is ready, but it looks fragile, not sure it will pass code review. However I can't find a better way so far, so I'm going to start review process.

[Mahesh Sawaiker (mahesh-sawaiker)]

Has this been fixed?

[Alexey Miroshkin (amirosh)]

Not yet, I found why tests pass fine while this bug exists. I opened a separate bug https://bugs.launchpad.net/keystone/+bug/1478000 and submitted a fix https://review.openstack.org/#/c/205667/
When that bug will be closed I'll send fix for this bug to review.

[Dolph Mathews (dolph)]

This appears to be fixed by https://review.openstack.org/#/c/206496/

[Alexey Miroshkin (amirosh)]

When I started working on this bug I was surprised that it wasn't detected by tests. So I checked tests and found several problems, https://review.openstack.org/#/c/206496/ was one of them, in fact it were 3-4 bugs in tests. So this one doesn't fix this bug but enables tests to detect this one. The main problem is that we have one V3 app (public) for admin and public endpoints, but we have to return the right endpoint in href. So far the best solution is to check the request's port.

[Yang Zhang (bjzyang)]

Hi Alexey, glad to know that the root cause was found, thanks for your help on this.
Btw, could you also merge the code changes into Juno & Kilo branch when the solution is ready for master?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/208168

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/226464

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226464
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=40c3942c12d1dd2c826d836987616838a73a64a1
Submitter: Jenkins
Branch: master

commit 40c3942c12d1dd2c826d836987616838a73a64a1
Author: Julien Danjou <email address hidden>
Date: Mon Sep 21 17:27:07 2015 +0200

    wsgi: fix base_url finding

    The current wsgi.Application.base_url() function does not work correctly
    if Keystone runs on something like "http://1.2.3.4/identity" which is now
    a default in devstack.

    This patch fixes that by using wsgiref.util to parse environment
    variable set in WSGI mode to find the real base url and returns the
    correct URL. The following environment variables will be used to
    produce the effective base url:

      HTTP_HOST
      SERVER_NAME
      SERVER_PORT
      SCRIPT_NAME

    Closes-Bug: #1381961
    Change-Id: I111c206a8a751ed117c6869f55f8236b29ab88a2

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b3 development milestone.

[Adam Young (ayoung)]

Reported in a downstream distribution that should have synced from this code as still a bug. please reconfirm.

[Adam Young (ayoung)]

This appears to be the necessary change

$ diff /usr/share/openstack-tripleo-heat-templates/puppet/services/keystone.yaml.orig /usr/share/openstack-tripleo-heat-templates/puppet/services/keystone.yaml
122d121
< keystone::public_endpoint: {get_param: [EndpointMap, KeystonePublic, uri_no_suffix]}

As it prevents the public_endpoint value from being set in the Keystone config file.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/368969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/368969
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=09f569b6a60e6426102212e650b490ea0dd5084f
Submitter: Jenkins
Branch: master

commit 09f569b6a60e6426102212e650b490ea0dd5084f
Author: Adam Young <email address hidden>
Date: Mon Sep 12 12:43:39 2016 -0400

    Unset Keystone public_endpoint

    The keystone public_endpoint value should be deduced from the calling
    request and not hardcoded, or it makes network isolation impossible.

    Change-Id: Ide6a65aa9393cb84591b0015ec5966cc01ffbcf8
    Closes-Bug: 1381961

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/208168
Reason: https://review.openstack.org/#/c/226464/ fixed the issue

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 2.2.0 release.