Public-routed net should be used for AdminUrl endpoints. UX: Error "Unable to establish connection to http" using user/pass credentials with cli client from a remote machine

[Ki]

Hi,

Latest version of mirantis/fuel, https://software.mirantis.com/quick-start/ followed, nodes up and running, instances can be created etc.

We use a machine RedHat with latest clients and connected to 172.16. network.

Env:
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_TENANT_NAME=admin
export OS_AUTH_URL="http://172.16.0.2:5000/v2.0/"

Does not allow to run certain commands against keystone:

$ keystone --debug user-list
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://172.16.0.2:5000/v2.0/tokens -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.0.2
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3058
DEBUG:keystoneclient.session:RESP: [200] {'date': 'Thu, 28 Aug 2014 10:48:10 GMT', 'content-type': 'application/json', 'content-length': '3058', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2014-08-28T10:48:10.397185", "expires": "2014-08-28T11:48:10Z", "id": "f7e3f6ea9909478eaa7f223a98403f30", "tenant": {"description": "admin tenant", "enabled": true, "id": "4b89376d32d942dd8b2682787a4d8450", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.1:8774/v2/4b89376d32d942dd8b2682787a4d8450", "region": "RegionOne", "internalURL": "http://192.168.0.1:8774/v2/4b89376d32d942dd8b2682787a4d8450", "id": "2771b8033d7c48e98b32cb8122115110", "publicURL": "http://172.16.0.2:8774/v2/4b89376d32d942dd8b2682787a4d8450"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.1:9696", "region": "RegionOne", "internalURL": "http://192.168.0.1:9696", "id": "43ecf8cab6f3483e819d15160b84d9b1", "publicURL": "http://172.16.0.2:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.0.1:9292", "region": "RegionOne", "internalURL": "http://192.168.0.1:9292", "id": "4c30ab87aa524f83b60fa4a6f0a7a418", "publicURL": "http://172.16.0.2:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.1:8777", "region": "RegionOne", "internalURL": "http://192.168.0.1:8777", "id": "251618fbbd8a4037a7af44e3bdee8b58", "publicURL": "http://172.16.0.2:8777"}], "endpoints_links": [], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http://192.168.0.1:8776/v1/4b89376d32d942dd8b2682787a4d8450", "region": "RegionOne", "internalURL": "http://192.168.0.1:8776/v1/4b89376d32d942dd8b2682787a4d8450", "id": "a6f59b9ca8d5449988df4295b4aa199b", "publicURL": "http://172.16.0.2:8776/v1/4b89376d32d942dd8b2682787a4d8450"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.1:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.1:8773/services/Cloud", "id": "44b528d6b7b44e1f809b7516586214d0", "publicURL": "http://172.16.0.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.1:8004/v1/4b89376d32d942dd8b2682787a4d8450", "region": "RegionOne", "internalURL": "http://192.168.0.1:8004/v1/4b89376d32d942dd8b2682787a4d8450", "id": "6aeade3902064f8b88e7521b6f534c53", "publicURL": "http://172.16.0.2:8004/v1/4b89376d32d942dd8b2682787a4d8450"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.1:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.1:5000/v2.0", "id": "2bf12e387cf24bd19e84b8dffe1a20fc", "publicURL": "http://172.16.0.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "66da4c2bf8d640e1817bc17344d19806", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "ef294954a9374443831bc909c4d94e78"]}}}

DEBUG:iso8601.iso8601:Parsed 2014-08-28T11:48:10Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'11', 'daydash': u'28', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'10', 'tz_minute': None, 'year': u'2014', 'separator': u'T', 'monthdash': u'08', 'day': None, 'minute': u'48'} with default timezone <iso8601.iso8601.Utc object at 0x7d8b90>
DEBUG:iso8601.iso8601:Got u'2014' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'08' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 8 for 'month' with default 8
DEBUG:iso8601.iso8601:Got u'28' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 28 for 'day' with default 28
DEBUG:iso8601.iso8601:Got u'11' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'48' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'10' for 'second' with default None
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.1:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: f7e3f6ea9909478eaa7f223a98403f30"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.1
Unable to establish connection to http://192.168.0.1:35357/v2.0/users

>> Even though the first REQ is ok against 172.16., there is a second request at 192.168, that is not public endpoint, thus cannot be reached.

But it will work OK with other commands, like keystone catalog, discover, token-get ...

--------------------------------
If, instead of using credentials, we use the admin token:
export SERVICE_TOKEN=J7frMTFf
export SERVICE_ENDPOINT=http://172.16.0.2:35357/v2.0/

Commands that worked before now fail:
[root@ms1 ~]# keystone token-get
'NoneType' object has no attribute 'has_service_catalog'

[root@ms1 ~]# keystone --debug discover
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://localhost:35357 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): localhost
ERROR:keystoneclient.generic.client:Unable to establish connection to http://localhost:35357
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystoneclient/generic/client.py", line 88, in _check_keystone_versions
    'application/json'})
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 563, in request
    resp = super(HTTPClient, self).request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/baseclient.py", line 21, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 324, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/session.py", line 260, in request
    resp = self._send_request(url, method, redirect, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/session.py", line 294, in _send_request
    raise exceptions.ConnectionRefused(msg)
ConnectionRefused: Unable to establish connection to http://localhost:35357
DEBUG:keystoneclient.session:REQ: curl -i -X GET https://localhost:35357 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): localhost
ERROR:keystoneclient.generic.client:Unable to establish connection to https://localhost:35357
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystoneclient/generic/client.py", line 88, in _check_keystone_versions
    'application/json'})
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 563, in request
    resp = super(HTTPClient, self).request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/baseclient.py", line 21, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 324, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/session.py", line 260, in request
    resp = self._send_request(url, method, redirect, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/session.py", line 294, in _send_request
    raise exceptions.ConnectionRefused(msg)
ConnectionRefused: Unable to establish connection to https://localhost:35357
No Keystone-compatible endpoint found

But, one of the commands that did not work before (user-list), now it works:

[root@ms1 ~]# keystone --debug user-list
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://172.16.0.2:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: J7frMTFf"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.0.2
DEBUG:urllib3.connectionpool:"GET /v2.0/users HTTP/1.1" 200 1387
DEBUG:keystoneclient.session:RESP: [200] {'date': 'Thu, 28 Aug 2014 14:03:48 GMT', 'content-type': 'application/json', 'content-length': '1387', 'vary': 'X-Auth-Token'}
RESP BODY: {"users": [{"username": "heat", "name": "heat", "id": "24911c43d0654280b27f7cf4e5f687b5", "enabled": true, "email": "<email address hidden>", "tenantId": "27bf657e0bc942a1bdaff162675fdc31"}, {"username": "cinder", "name": "cinder", "id": "434b810aaeea449aabed39a56ef48245", "enabled": true, "email": "cinder@localhost", "tenantId": "27bf657e0bc942a1bdaff162675fdc31"}, {"username": "ceilometer", "name": "ceilometer", "id": "468215ef59854a97bd1a9364858fd78c", "enabled": true, "email": "ceilometer@localhost", "tenantId": "27bf657e0bc942a1bdaff162675fdc31"}, {"username": "glance", "name": "glance", "id": "5def323031d2428fbe0748ae71aafa98", "enabled": true, "email": "glance@localhost", "tenantId": "27bf657e0bc942a1bdaff162675fdc31"}, {"username": "admin", "name": "admin", "id": "66da4c2bf8d640e1817bc17344d19806", "enabled": true, "email": "<email address hidden>", "tenantId": "4b89376d32d942dd8b2682787a4d8450"}, {"username": "nova", "name": "nova", "id": "6fd1b5d0b7db41cca56ef155f8f58773", "enabled": true, "email": "nova@localhost", "tenantId": "27bf657e0bc942a1bdaff162675fdc31"}, {"username": "neutron", "name": "neutron", "id": "a70cc4723eb9468d8fc16292b1d81360", "enabled": true, "email": "neutron@localhost", "tenantId": "27bf657e0bc942a1bdaff162675fdc31"}, {"username": "litp", "name": "litp", "enabled": true, "email": "<email address hidden>", "id": "c21f0baccfaa49668d05c64e739a3753"}]}

+----------------------------------+------------+---------+----------------------+
| id | name | enabled | email |
+----------------------------------+------------+---------+----------------------+
| 66da4c2bf8d640e1817bc17344d19806 | admin | True | <email address hidden> |
| 468215ef59854a97bd1a9364858fd78c | ceilometer | True | ceilometer@localhost |
| 434b810aaeea449aabed39a56ef48245 | cinder | True | cinder@localhost |
| 5def323031d2428fbe0748ae71aafa98 | glance | True | glance@localhost |
| 24911c43d0654280b27f7cf4e5f687b5 | heat | True | <email address hidden> |
| c21f0baccfaa49668d05c64e739a3753 | litp | True | <email address hidden> |
| a70cc4723eb9468d8fc16292b1d81360 | neutron | True | neutron@localhost |
| 6fd1b5d0b7db41cca56ef155f8f58773 | nova | True | nova@localhost |
+----------------------------------+------------+---------+----------------------+

--------------------------------
Last, setting both credentials and service_token:
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_TENANT_NAME=admin
export OS_AUTH_URL="http://172.16.0.2:5000/v2.0/"
export SERVICE_TOKEN=J7frMTFf
export SERVICE_ENDPOINT=http://172.16.0.2:35357/v2.0/

Will make non-keystone services work (nova, cinder) but some kyestone will still not work:
[root@ms1 ~]# keystone catalog
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
'NoneType' object has no attribute 'has_service_catalog'

We have 3 different mirantis/fuel deployment, with variations in networking, and we see similar behaviour in all of them.
Somehow the admin credentials are not working with keystone using remote clients.
Workaround is to keep switching between credentials/admin_token, depending on the cli.
Related bug opened against keystone: https://bugs.launchpad.net/keystone/+bug/1362630

Thanks!!!

Endpoints:
[root@node-1 tmp]# keystone endpoint-list
+----------------------------------+-----------+-----------------------------------------+------------------------------------------+------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+-----------------------------------------+------------------------------------------+------------------------------------------+----------------------------------+
| 3109aec7633f4def8d33f558522a2bb6 | RegionOne | http://172.16.0.2:8004/v1/%(tenant_id)s | http://192.168.0.1:8004/v1/%(tenant_id)s | http://192.168.0.1:8004/v1/%(tenant_id)s | a605cbfa1f3248198a309f94f67ac8a8 |
| 5054e540d48241569be9b193135b9693 | RegionOne | http://172.16.0.2:9292 | http://192.168.0.1:9292 | http://192.168.0.1:9292 | d7599225d26a403e9e738d909b5b4726 |
| 6e0d9794a25f466d97a4607327d19673 | RegionOne | http://172.16.0.2:8777 | http://192.168.0.1:8777 | http://192.168.0.1:8777 | cedee33840bb457e94cacdef8e167d12 |
| 726e6479f8fa483180153adfa235cc1c | RegionOne | http://172.16.0.2:5000/v2.0 | http://192.168.0.1:5000/v2.0 | http://192.168.0.1:35357/v2.0 | 39af888b0ae840b0865490776eaf18e3 |
| 7a87dd44b470484a925f2c9d3a3b993a | RegionOne | http://172.16.0.2:8774/v2/%(tenant_id)s | http://192.168.0.1:8774/v2/%(tenant_id)s | http://192.168.0.1:8774/v2/%(tenant_id)s | e756e858792849fbbfc2de5d67eae361 |
| ac8cc4170c5948feb9fdb5c2555dc473 | RegionOne | http://172.16.0.2:9696 | http://192.168.0.1:9696 | http://192.168.0.1:9696 | 8b2ed390b10b47db9e82f0feffafa266 |
| db61321b69154407817c74801334c2f8 | RegionOne | http://172.16.0.2:8773/services/Cloud | http://192.168.0.1:8773/services/Cloud | http://192.168.0.1:8773/services/Admin | a681dcf28a644ec9b0875892723644a7 |
| e47c4c41210e495b84e0d42a00c1796d | RegionOne | http://172.16.0.2:8776/v1/%(tenant_id)s | http://192.168.0.1:8776/v1/%(tenant_id)s | http://192.168.0.1:8776/v1/%(tenant_id)s | 3575d428a77843a9aa0a8e45db800a79 |
+----------------------------------+-----------+-----------------------------------------+------------------------------------------+------------------------------------------+----------------------------------+

[Sergey Vasilenko (xenolog)]

as I remember, part of keystone calls can be executed only through administrative keystone port (35357).

[Ki (kieleth)]

Thanks Sergey,

Apologies for the machine-gun questions:

It is that by design?

Do you know any piece of documentation you can refer me to about it?

Shouldn't a external client be able to just authenticate against keystone, and then OS do internally the other's services requests, even if they are admin?

"hacking" the mysql to change the admin endpoint to a public ip then is then the actual "solution" for this?

[Ryan Moe (rmoe)]

Setting the admin URL to the internal network was an intentional design decision. If you need access to the admin URL from outside of the management network then updating the endpoint is probably the best way to accomplish this.

[Ki (kieleth)]

Hey Ryan,

Thanks for response.

I'm a newcomer to Openstack, but shouldn't be this documented somewhere?

if yes, can you point me where?

If not, I think its important and should be available for others, where should we put it? I can take that action if you agree.

Regards,

Luis

[Mike Scherbakov (mihgen)]

There is some information on the topic: http://docs.openstack.org/security-guide/content/ch021_paste-and-middleware.html. I believe we should add something to our docs, so users don't get stuck with this.

[Ki (kieleth)]

Thanks Mike.
There's a linked bug to this, that aims to explain why the clients target admin endpoints, even though they authenticate against public keystone's.
For reference and follow-up:
https://bugs.launchpad.net/keystone/+bug/1362630

[Mike Scherbakov (mihgen)]

Thanks, Ki, we will get a closer look into this.
I marked this bug as affecting 6.0. I think we should come with whether fix or documentation which clarifies everything around it.

[Alexander Makarov (amakarov)]

In this document:
http://developer.openstack.org/api-ref-identity-v2.html#os-ksadm-admin-ext
API call "/v2.0/users" explicitly stated as a part of "OS-KSADM admin extension"

[Bogdan Dobrelya (bogdando)]

Reopening as there is confirmed duplicate https://bugs.launchpad.net/fuel/+bug/1430309

[Bogdan Dobrelya (bogdando)]

As it figured out in duplicating bug, the non admin (public) URL is affected by this issue as well. At some point, see http://paste.openstack.org/show/203957/ line #62, keystone client contacts the *admin URL* via internal IP and fails, if it is not reachable from outside (and it is normally not reachable)

[Bogdan Dobrelya (bogdando)]

I'd say this is a keystone client bug as it should never contact admin URL unless it has been told to do so!

[Bogdan Dobrelya (bogdando)]

Here is the way how it works:

first get a token by your creds
 /usr/bin/keystone --debug token-get
second, get a public endpoint
 /usr/bin/keystone --debug endpoint-get --service identity
next use this endpoint and the given token to query keystone, for example
 /usr/bin/keystone --debug --os-token=7dfe9dc1914b4a8b91ba44ab47cae639 --os-endpoint=http://10.109.1.2:5000/v2.0/ tenant-list

[Bogdan Dobrelya (bogdando)]

I set it to triaged as there is a work around.
I'm not sure is it the correct way to use keystone client, but at least it works

[Bogdan Dobrelya (bogdando)]

IMHO, that is how it works by design: when using token-less auth, perhaps it is assumed that you are in trusted network, hence admin endpoint must be contacted in order to verify your creds (since you don't specify the token). But there is a way to get a token from "untrusted locations" w/o the possibility to contact admin endpoint and use it for other requests. And this way described above, in the comment #12.

Sounds logical, as for me. Not sure if there are official docs for this

[Alexander Makarov (amakarov)]

This beravior is expected: public and admin interfaces were separated intentionally. I'd ask fuel team if they can do something to it.

[Bogdan Dobrelya (bogdando)]

@Alexandr, Fuel library team cannot modify the keystone client. Please read the comments above. The issue is that there is some point exist when keystone client tries to communicate with ADMIN endpoint. This happens when user is being authenticated only with creds. And there is no such issue if some token was specified as an argument. Fuel team can do nothing here as it looks like some keystone client built-in logic.

[Bogdan Dobrelya (bogdando)]

Note, make the admin endpoint exposed publicly is not an option as well.

[Bogdan Dobrelya (bogdando)]

Lowering back to medium as this does not impact the cloud operations and there is a work around described

[Thomas Hambleton (tomhambleton)]

The work around described above does not work for commands like tenant-create, user-list, user-create, etc. The only way I see to issue these commands remotely is to make the management network externally routable. However, the management network (in Fuel) is supposed to be private. The only way we can issues these commands now is by running them on the controller.

Also, if the management network is supposed to be externally routable then Fuel needs to allow the address range to be specified to support VRRP.

Please raise back to High.

[Boris Bobrov (bbobrov)]

This is not something we, MOS-Keystone, can fix. Keystoneclient uses endpoints from keystone, which point to management network. If you need other behavior, endpoints need to be set appropriately.

[Bogdan Dobrelya (bogdando)]

Folks, the management network is supposed to be private in every security aware deployment, by design. It contains too much sensitive traffic to be exposed outside.

[Bogdan Dobrelya (bogdando)]

@Thomas, you're right - those commands like user-list will not work with the method described in https://bugs.launchpad.net/mos/+bug/1362641/comments/12

[Bogdan Dobrelya (bogdando)]

AFAIKT from this article https://linuxacademy.com/blog/linux/understanding-keystone-endpoints/ the issue is that Fuel configures admin url via publicly unreachable management network VIP. This is an issue, as this makes impossible to make admin requests from outside. And this bug is exactly such a request - to allow admin actions being performed from outside.

The solution is:
A) either to use the public network VIP for admin url endpoint
B) or to introduce another type of the Openstack Admin API network for Fuel

the latter one seems an overkill as the public network should allow to do the same things with less changes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/213080

[Vladimir Kuklin (vkuklin)]

Folks

Fixing this bug actually introduces major security flaw as we do not currently want anyone from outside to talk to admin endpoint. Actually this can be fixed by providing some kind of secure tunnel to the one who is running a client. Also, I would like to mention that change like that actually affects reference architecture and can in no way be accepted after Code Freeze is declared.

[Bogdan Dobrelya (bogdando)]

Moved to the 8.0 according to the Vladimir's comment and review

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/213080
Reason: Moved to 8.0

[Bogdan Dobrelya (bogdando)]

I'm wondering if this issue could be related to how Fuel configures the auth_uri in the api-paste.ini files

See http://docs.openstack.org/havana/install-guide/install/apt/content/neutron-install.dedicated-network-node.html.
Seems obsolete, but the warning looks like exactly the case: "keystoneclient.middleware.auth_token: You must configure auth_uri to point to the public identity endpoint. Otherwise, clients might not be able to authenticate against an admin endpoint."

[Bogdan Dobrelya (bogdando)]

Here is the conf example:
root@node-1:~# hiera management_vip
10.109.2.3
root@node-1:~# rgrep auth_uri /etc/nova
/etc/nova/nova.conf:auth_uri=http://10.109.2.3:5000/
root@node-1:~# rgrep auth_uri /etc/neutron
/etc/neutron/neutron.conf:auth_uri = http://10.109.2.3:35357/v2.0
/etc/neutron/api-paste.ini:auth_uri=http://10.109.2.3:35357/v2.0
root@node-1:~# rgrep auth_uri /etc/glance
/etc/glance/glance-registry.conf:auth_uri=http://10.109.2.3:5000/
/etc/glance/glance-api.conf:auth_uri=http://10.109.2.3:5000/
root@node-1:~# rgrep auth_uri /etc/cinder
/etc/cinder/api-paste.ini:auth_uri=http://10.109.2.3:5000/
/etc/cinder/cinder.conf:auth_uri=http://10.109.2.3:5000/

The neutron's auth_uri uses port 35357, while the others use 5000. Although, this looks nothing to have to the keystone CLI issues in this bug.

[Stanislaw Bogatkin (sbogatkin)]

Bogdan is right. We shouldn't expose admin endpoints as public. It's violate security access.

[Adam Heczko (aheczko-mirantis)]

Folks, could anyone of you describe why Keystone exposes its API on two separate TCP ports? This question is intriguing me for some time and I don't understand this design decision. Usually when one wants to secure some API operations, it does so on the API level and not on the TCP layer. IMO Keystone API endpoints behavior should be consistent with expected by keystone-openstackclient , and clearly some CLI operations are filing down because of this misunderstanding / misconfiguration. I don't see much benefit in current TCP port / IP address separation approach. TL;DR: IMO both Keystone API endpoints should be available on common IP address.

[Stanislaw Bogatkin (sbogatkin)]

Adam, it was bad design of keystone itself, as I see it. Some actions can be done only by accessing admin port. It was rewritten and for Keystone v3 API now you can use 5000 and 35357 equally and use policies to restrict access for users.

[Boris Bobrov (bbobrov)]

Stanislaw is right. This was fixed in v3. Now there are talks even about moving to port 80.

[Castulo J. Martinez (castulo-martinez)]

I have a question Guys, if this is not going to get fixed then there is no way to run Tempest tests in an Open Stack deployed by Fuel, at least not in parallel since for that you would need to configure Tempest to run with "tenant isolation" and for this Tempest needs to create a tenant and a user for each test, which fails because it cannot reach the adminURL. There has to be a workaround to be able to run Tempest tests in prallel in a fuel deployment.