Unintended endpoint selects when issuing Keystone API

[Hiroo Kitamura]

Tacker uses "keystoneauth" internally when issuing Tacker-API, but it detected an event that seems to be a bug.
I issued a bugreport (#1) in Tacker, but I found a problem with keystoneauth by additional investigation, so I reported it.

# 1: Bugreport in Tacker
https://bugs.launchpad.net/tacker/+bug/1920088

Details are shown below.

* Preparation
  a) Change admin endpoint url to inaccessible IP address.
$ openstack endpoint set 85ef9c98d7394114a96209cf1cd61078 --url http://192.168.2.101/identity

  b) Modify admin endpoint url.
$ openstack endpoint list --service keystone
  "+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------+
  | ID | Region | Service Name | Service Type | Enabled | Interface | URL |
  +----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------+
  | 85ef9c98d7394114a96209cf1cd61078 | RegionOne | keystone | identity | True | admin | http://192.168.2.101/identity |
  | b8ce50d5b7c34f8491643554654a8236 | RegionOne | keystone | identity | True | public | http://172.22.39.3/identity |
  +----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------+

  c) When I executs Tacker-API, it fails.
$ openstack vim register --config-file /opt/stack/data/tacker/vim_config.yaml VTEST2
  Request Failed: internal server error while processing your request.

* Related logs
 2021-05-24 01:54:07.742 DEBUG tacker.wsgi [-] (27696) accepted ('172.22.39.3', 43944) from (pid=27696) server /usr/local/lib/python3.6/dist-packages/eventlet/wsgi.py:992
  2021-05-24 01:54:07.746 DEBUG tacker.alarm_receiver [-] Process request: POST /v1.0/vims.json HTTP/1.0
  Accept: application/json
  Accept-Encoding: gzip, deflate
  Connection: keep-alive
  Content-Length: 296
  Content-Type: application/json
  Host: 172.22.39.3:9890
  User-Agent: python-tackerclient
  X-Auth-Token: gAAAAABgqwc_WYsN455qcZgtkTZJqaSR2ScAuLuaZvF19Y91GxUBXwxdVaUjbwJTcUli8E4ENaJr9ago8BTSTdkwKw5xBRglFIVV9GfK7o7yIeHTwMjNrWRw_dmRHwbBb0WVisxxiG9s3cy-arMkLfaJL1b2rehEXcDfKodDjltAOwZjA3SEbHE

  {"vim": {"auth_url": "http://172.22.39.3/identity", "type": "openstack", "vim_project": {"name": "nfv", "project_domain_name": "Default"}, "auth_cred": {"username": "nfv_user", "password": "***", "user_domain_name": "Default", "cert_verify": "False"}, "name": "VTEST2", "is_default": false}} from (pid=27696) process_request /opt/stack/tacker/tacker/alarm_receiver.py:52
  2021-05-24 01:54:07.751 DEBUG keystoneauth1.session [-] http://172.22.39.3/identity/v3/ from (pid=27696) request /usr/local/lib/python3.6/dist-packages/keystoneauth1/session.py:816
  2021-05-24 01:54:07.751 DEBUG keystoneauth1.session [-] http://172.22.39.3/identity/v3/auth/tokens from (pid=27696) request /usr/local/lib/python3.6/dist-packages/keystoneauth1/session.py:822
  ** 2021-05-24 01:54:07.752 DEBUG keystoneauth1.session [-] http://172.22.39.3/identity/v3/auth/tokens from (pid=27696) request /usr/local/lib/python3.6/dist-packages/keystoneauth1/session.py:900 **

* Investigation result
The following response-body is received in "2021-05-24 01: 54: 07.75 DEBUG keystoneauth1.session [-] http://172.22.39.3/identity/v3/auth/tokens from (pid = 27696) request /usr/local/lib/python3.6/dist-packages/keystoneauth1/session.py:900".
The endpoints field of the response-body contains all endpoint information including public/admin/internal.
I assume that the appropriate endpoint selection is not done in this response-body reception processing in keystoneauth.
As a result, it is assumed that the admin endpoint is selected and an error is encountered.

* Response-body
curl -g -i -X POST http://172.22.39.3/identity/v3/auth/tokens -d '{"auth":{"identity":{"methods":["password"],"password":{"user":{"domain":{"name":"Default"},"name":"jftt_test","password":"jftt_test"}}},"scope":{"project":{"domain":{"name":"Default"},"name":"project_a"}}}}' -H 'Content-type:application/json'

{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "ee1c6495f0eb42849f825f964dd7a700", "name": "nfv_user", "password_expires_at": null}, "audit_ids": ["EJcT1ZcFTxGSZzUb4oe-vw"], "expires_at": "2021-05-22T07:22:39.000000Z", "issued_at": "2021-05-21T07:22:39.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55bb83431b9d490abf25c327bf0704de", "name": "nfv"}, "is_domain": false, "roles": [{"id": "b92c52e2a4da4afd960fe55c37a28552", "name": "member"}, {"id": "bdb8b73e986b433baafb95c1fc7167c1", "name": "admin"}, {"id": "ff57475250734f7dbd028e23e58ba5ec", "name": "reader"}], "catalog": [{"endpoints": [{"id": "85ef9c98d7394114a96209cf1cd61078", "interface": "admin", "region_id": "RegionOne", "url": "http://192.168.2.101/identity", "region": "RegionOne"}, {"id": "b8ce50d5b7c34f8491643554654a8236", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/identity", "region": "RegionOne"}], "id": "03f3bae643934ba9a606220799378f30", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "d4997e90caf448eb808ba284d5ab1ac4", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/compute/v2/55bb83431b9d490abf25c327bf0704de", "region": "RegionOne"}], "id": "0d55fb1751844784b4c6070138201fd5", "type": "compute_legacy", "name": "nova_legacy"}, {"endpoints": [{"id": "4f1980dda0194daab091abf0ea211d53", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3:9890/", "region": "RegionOne"}, {"id": "78393538068b4fe18ad8bb55ce9e98fd", "interface": "internal", "region_id": "RegionOne", "url": "http://172.22.39.3:9890/", "region": "RegionOne"}, {"id": "a031a939c862407cbb17a4855453e601", "interface": "admin", "region_id": "RegionOne", "url": "http://172.22.39.3:9890/", "region": "RegionOne"}], "id": "22f1c1aa5064459caf6dc07ce77732ce", "type": "nfv-orchestration", "name": "tacker"}, {"endpoints": [{"id": "79edc53b64594423a3e5b996423c5fe4", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/key-manager", "region": "RegionOne"}, {"id": "81880ebdb3e04c9494be7847b67cb5a0", "interface": "internal", "region_id": "RegionOne", "url": "http://172.22.39.3/key-manager", "region": "RegionOne"}], "id": "39e31f98b099499c9c42693a672dda7c", "type": "key-manager", "name": "barbican"}, {"endpoints": [{"id": "555c013e9b2d49509d1b896b71e354f5", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/heat-api-cfn/v1", "region": "RegionOne"}], "id": "45c873aaaac24b93a5d19251f1e29a26", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "2648f7fe84284d8097ef3c132d641c26", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/volume/v2/55bb83431b9d490abf25c327bf0704de", "region": "RegionOne"}], "id": "4ffdeef514d548789c06e980d0175b98", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "21fa9cd712e843dba6e627ef0045de4d", "interface": "internal", "region_id": "RegionOne", "url": "http://172.22.39.3:5000/v3", "region": "RegionOne"}, {"id": "5392ac73f1a34487b3bdcec96b4e36d1", "interface": "admin", "region_id": "RegionOne", "url": "http://172.22.39.3:35357/v3", "region": "RegionOne"}, {"id": "c618cc6f79ae4ab3a845f6ab646d53fc", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3:5000/v3", "region": "RegionOne"}], "id": "5d75820e41774f39ae4584b1722b9d16", "type": "identityv3", "name": "keystonev3"}, {"endpoints": [{"id": "42a8fb9903ea4fdc81f46735f3f38e9f", "interface": "internal", "region_id": "RegionOne", "url": "http://172.22.39.3:8042", "region": "RegionOne"}, {"id": "c9b94436ca1147c4a3c609d4b476b523", "interface": "admin", "region_id": "RegionOne", "url": "http://172.22.39.3:8042", "region": "RegionOne"}, {"id": "f46d1ecfd693469183c4441fe81e0403", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3:8042", "region": "RegionOne"}], "id": "73f80c8c1640426d8d7010248924786f", "type": "alarming", "name": "aodh"}, {"endpoints": [{"id": "b0d85bed063f4e2897d732b0c68e7281", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3:9696/", "region": "RegionOne"}], "id": "ab4ece1e08974e69ad721bf5085bcfcf", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0254babd973c4300b823620187106b6f", "interface": "internal", "region_id": "RegionOne", "url": "http://172.22.39.3:8989/v2", "region": "RegionOne"}, {"id": "447e5bbbe7e547ce8242e21afaa2f04a", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3:8989/v2", "region": "RegionOne"}, {"id": "dad87c22d8dc435eb4387f4652d6e8de", "interface": "admin", "region_id": "RegionOne", "url": "http://172.22.39.3:8989/v2", "region": "RegionOne"}], "id": "b304ec974d934505a2d416df0c23e14a", "type": "workflowv2", "name": "mistral"}, {"endpoints": [{"id": "71edf1357dc846b9ad22a3e96ce8aa99", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/compute/v2.1", "region": "RegionOne"}], "id": "b9e90607da894536b0125df6a1023391", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "00d6b4a63494414d9de0f5c4edf4d2fd", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/image", "region": "RegionOne"}], "id": "d5a0b7f84fa94002914e18514a8e4216", "type": "image", "name": "glance"}, {"endpoints": [{"id": "158085dd85ac4bdda66809581bcec53c", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/heat-api/v1/55bb83431b9d490abf25c327bf0704de", "region": "RegionOne"}], "id": "e04305f5ef7e49f39ac277d9ff5902db", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "d16a2b6273e84c28b0403b03cfb2280d", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/volume/v3/55bb83431b9d490abf25c327bf0704de", "region": "RegionOne"}], "id": "e8cae1af8dcc42ada5df3670d796f4c5", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "184a7c3514dc4200ac8be1a497fbe2e2", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/reservation/v1", "region": "RegionOne"}], "id": "f292ff7fbbb7446980ac2bf371ed7cbe", "type": "reservation", "name": "blazar"}, {"endpoints": [{"id": "bc442df0d3414a558f50b74be5059225", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/volume/v3/55bb83431b9d490abf25c327bf0704de", "region": "RegionOne"}], "id": "f7d842e49ae342cbadfc8126a47a9db0", "type": "block-storage", "name": "cinder"}, {"endpoints": [{"id": "3bc41234676144fb87db5839c4686af5", "interface": "public", "region_id": "RegionOne", "url": "http://172.22.39.3/placement", "region": "RegionOne"}], "id": "f817a23d0ffe4e7d98ae338839578c50", "type": "placement", "name": "placement"}]}}

[Hiromu Asahina (h-asahina)]

I think it might be a keystone client problem.
The client doesn't use `auth_url` set by `session`.

>>> from keystoneauth1.identity import v3
>>> from keystoneauth1 import session
>>> from keystoneclient import client
>>> os_auth_url
'http://127.0.0.1/identity/v3'
>>> v3_auth = v3.Password(auth_url = os_auth_url, username = os_username,
... password = os_password, project_name = os_tenant_name,
... project_domain_name = 'default', user_domain_name = 'default')
>>> v3_auth.auth_url
'http://127.0.0.1/identity/v3'
>>> ses = session.Session(auth=v3_auth, verify=True)
>>> cli = client.Client('v3', session=ses)
>>> cli.auth_url
'http://192.168.33.14/identity/v3'
>>> cli.session.auth.auth_url
'http://127.0.0.1/identity/v3'