user management does not work (regression from keystoneclient)

[gustavo panizzo]

Hello
     In my production clouds I cannot do use openstack command to list or manage users it always fails, the example I show is with 1.6.0 from pip but I discovered it when it was reported to python-openstackclient Debian package

(venv)root@sid:~# openstack --insecure user list
WARNING: keystoneclient.auth.identity.generic.base Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
ERROR: openstack
(venv)root@sid:~# openstack --insecure --debug user list
DEBUG: openstackclient.shell options: Namespace(access_token_endpoint='', auth_type='', auth_url='https://keystone.lan.example.com:35357/v2.0', cacert='', client_id='', client_secret='', cloud='', debug=True, default_domain='default', deferred_help=False, domain_id='', domain_name='', endpoint='', identity_provider='', insecure=True, interface='', log_file=None, os_compute_api_version='', os_identity_api_version='', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', password='password', project_domain_id='', project_domain_name='', project_id='', project_name='admin', protocol='', region_name='zh-office-01', scope='', timing=False, token='', trust_id='', url='', user_domain_id='', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None)
DEBUG: openstackclient.shell defaults: {'auth_type': 'osc_password', 'compute_api_version': '2', 'database_api_version': '1.0', 'api_timeout': None, 'baremetal_api_version': '1', 'cacert': None, 'image_api_use_tasks': False, 'floating_ip_source': 'neutron', 'key': None, 'interface': None, 'network_api_version': '2', 'image_format': 'qcow2', 'object_api_version': '1', 'image_api_version': '1', 'verify': True, 'identity_api_version': '2', 'volume_api_version': '1', 'cert': None, 'secgroup_source': 'neutron', 'dns_api_version': '2', 'disable_vendor_agent': {}}
DEBUG: openstackclient.shell cloud cfg: {'auth_type': 'osc_password', 'compute_api_version': '2', 'database_api_version': '1.0', 'interface': None, 'network_api_version': '2', 'image_format': 'qcow2', 'object_api_version': '1', 'image_api_version': '1', 'verify': True, 'timing': False, 'dns_api_version': '2', 'verbose_level': 3, 'region_name': 'zh-office-01', 'insecure': True, 'api_timeout': None, 'baremetal_api_version': '1', 'auth': {'username': 'admin', 'tenant_name': 'admin', 'project_name': 'admin', 'password': 'password', 'auth_url': 'https://keystone.lan.example.com:35357/v2.0'}, 'default_domain': 'default', 'image_api_use_tasks': False, 'floating_ip_source': 'neutron', 'key': None, 'cacert': None, 'deferred_help': False, 'identity_api_version': '2', 'volume_api_version': '1', 'cert': None, 'secgroup_source': 'neutron', 'debug': True, 'disable_vendor_agent': {}}
DEBUG: openstackclient.shell compute API version 2, cmd group openstack.compute.v2
DEBUG: openstackclient.shell network API version 2, cmd group openstack.network.v2
DEBUG: openstackclient.shell image API version 1, cmd group openstack.image.v1
DEBUG: openstackclient.shell volume API version 1, cmd group openstack.volume.v1
DEBUG: openstackclient.shell identity API version 2, cmd group openstack.identity.v2
DEBUG: openstackclient.shell object_store API version 1, cmd group openstack.object_store.v1
INFO: openstackclient.shell command: user list -> openstackclient.identity.v2_0.user.ListUser
DEBUG: openstackclient.api.auth Auth plugin osc_password selected
DEBUG: openstackclient.api.auth auth_type: osc_password
INFO: openstackclient.common.clientmanager Using auth plugin: osc_password
DEBUG: openstackclient.common.clientmanager Using parameters {'username': 'admin', 'tenant_name': 'admin', 'password': 'password', 'project_name': 'admin', 'auth_url': 'https://keystone.lan.example.com:35357/v2.0'}
DEBUG: openstackclient.common.clientmanager Get auth_ref
DEBUG: keystoneclient.session REQ: curl -g -i --insecure -X GET https://keystone.lan.example.com:35357/v2.0 -H "Accept: application/json" -H "User-Agent: python-openstackclient"
INFO: requests.packages.urllib3.connectionpool Starting new HTTPS connection (1): keystone.lan.example.com
/root/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: requests.packages.urllib3.connectionpool "GET /v2.0 HTTP/1.1" 404 90
DEBUG: keystoneclient.session RESP: [404] content-length: 90 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.2.22 (Ubuntu) connection: Keep-Alive date: Fri, 11 Sep 2015 06:16:45 GMT content-type: application/json x-distribution: Ubuntu
RESP BODY: {"error": {"message": "Could not find version, v2.0.", "code": 404, "title": "Not Found"}}

DEBUG: keystoneclient.session Request returned failure status: 404
WARNING: keystoneclient.auth.identity.generic.base Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
DEBUG: keystoneclient.auth.identity.v2 Making authentication request to https://keystone.lan.example.com:35357/v2.0/tokens
/root/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: requests.packages.urllib3.connectionpool "POST /v2.0/tokens HTTP/1.1" 200 1915
DEBUG: openstackclient.identity.v2_0.user.ListUser take_action(Namespace(columns=[], formatter='table', long=False, max_width=0, project=None, quote_mode='nonnumeric'))
DEBUG: openstackclient.identity.client Instantiating identity client: <class 'openstackclient.identity.client.IdentityClientv2'>
DEBUG: keystoneclient.auth.identity.v2 Making authentication request to https://keystone.lan.example.com:35357/v2.0/tokens
/root/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: requests.packages.urllib3.connectionpool "POST /v2.0/tokens HTTP/1.1" 200 1915
DEBUG: keystoneclient.session REQ: curl -g -i --insecure -X GET https://keystone.lan.example.com:35357/ -H "Accept: application/json" -H "User-Agent: python-openstackclient"
/root/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: requests.packages.urllib3.connectionpool "GET / HTTP/1.1" 300 364
DEBUG: keystoneclient.session RESP: [300] content-length: 364 vary: X-Auth-Token keep-alive: timeout=5, max=97 server: Apache/2.2.22 (Ubuntu) connection: Keep-Alive date: Fri, 11 Sep 2015 06:16:45 GMT content-type: application/json x-distribution: Ubuntu
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2013-03-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v3+xml"}], "id": "v3.0", "links": [{"href": "https://keystone.lan.example.com:35357/v3/", "rel": "self"}]}]}}

ERROR: openstack
Traceback (most recent call last):
  File "/root/venv/local/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/root/venv/local/lib/python2.7/site-packages/cliff/display.py", line 92, in run
    column_names, data = self.take_action(parsed_args)
  File "/root/venv/local/lib/python2.7/site-packages/openstackclient/identity/v2_0/user.py", line 222, in take_action
    data = identity_client.users.list(tenant_id=project)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/v2_0/users.py", line 126, in list
    return self._list("/users%s" % query, "users")
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/base.py", line 124, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/openstackclient/common/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 328, in request
    raise exceptions.EndpointNotFound()
EndpointNotFound
DEBUG: openstackclient.shell clean_up ListUser:
ERROR: openstackclient.shell Traceback (most recent call last):
  File "/root/venv/local/lib/python2.7/site-packages/openstackclient/shell.py", line 152, in run
    return super(OpenStackShell, self).run(argv)
  File "/root/venv/local/lib/python2.7/site-packages/cliff/app.py", line 255, in run
    result = self.run_subcommand(remainder)
  File "/root/venv/local/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/root/venv/local/lib/python2.7/site-packages/cliff/display.py", line 92, in run
    column_names, data = self.take_action(parsed_args)
  File "/root/venv/local/lib/python2.7/site-packages/openstackclient/identity/v2_0/user.py", line 222, in take_action
    data = identity_client.users.list(tenant_id=project)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/v2_0/users.py", line 126, in list
    return self._list("/users%s" % query, "users")
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/base.py", line 124, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/openstackclient/common/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/root/venv/local/lib/python2.7/site-packages/keystoneclient/session.py", line 328, in request
    raise exceptions.EndpointNotFound()
EndpointNotFound

keystoneclient

(venv)root@sid:~# keystone --insecure --debug user-list
/root/venv/local/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/root/venv/local/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/root/venv/local/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/root/venv/local/lib/python2.7/site-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/root/venv/local/lib/python2.7/site-packages/keystoneclient/httpclient.py:376: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to https://keystone.lan.example.com:35357/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): keystone.lan.example.com
/root/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 1915
DEBUG:keystoneclient.session:REQ: curl -g -i --insecure -X GET https://keystone.lan.example.com:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b50d5cb815ae76b8397b25cf14344c6e47fc296f"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): keystone.lan.example.com
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0/users HTTP/1.1" 200 1709
DEBUG:keystoneclient.session:RESP: [200] content-length: 1709 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.2.22 (Ubuntu) connection: Keep-Alive date: Fri, 11 Sep 2015 06:20:17 GMT content-type: application/json x-distribution: Ubuntu
RESP BODY: {"users": [{"username": "nova", "name": "nova", "id": "490501e90c1047399de72403f088b2af", "enabled": true, "email": null, "tenantId": "041644e4326b49f1b7733ac766aeb776"}, {"username": "cmdb", "name": "cmdb", "id": "7299f9f6d5be4c9dbb615869dbeb846b", "enabled": true, "email": null, "tenantId": "3bfe4d05504140e38bba176ac8201575"}, {"username": "user1234567", "name": "user1234567", "id": "7c0ddd1e33384da683588ad49e1afdd1", "enabled": true, "email": null, "tenantId": "32669baf5beb4eed812212dae7ee0182"}, {"username": "nova-notify", "name": "nova-notify", "id": "8f9d271dd37c458e94ba6f9e6e20a786", "enabled": true, "email": null, "tenantId": "041644e4326b49f1b7733ac766aeb776"}, {"username": "neutron-admin", "name": "neutron-admin", "id": "a99d5a80c19a465ca7c46712152fb410", "enabled": true, "email": null, "tenantId": "041644e4326b49f1b7733ac766aeb776"}, {"username": "neutron-nova", "name": "neutron-nova", "id": "bb9394c596bc44cc82fb87da86c5f1de", "enabled": true, "email": null, "tenantId": "041644e4326b49f1b7733ac766aeb776"}, {"username": "user123456", "name": "user123456", "id": "e237c9cb2bca4c25a570e4c21f0af906", "enabled": true, "email": "<email address hidden>", "tenantId": "32669baf5beb4eed812212dae7ee0182"}, {"username": "glance", "name": "glance", "id": "e3477d7cf02749b1b505d2fa5b5d6ede", "enabled": true, "email": null, "tenantId": "041644e4326b49f1b7733ac766aeb776"}, {"username": "neutron", "name": "neutron", "id": "edf47979c2ce4931b1b9a2d1f9859931", "enabled": true, "email": null, "tenantId": "041644e4326b49f1b7733ac766aeb776"}, {"username": "admin", "name": "admin", "id": "fdc9314674ae49e797b4258226c20009", "enabled": true, "email": null, "tenantId": "32669baf5beb4eed812212dae7ee0182"}]}

+----------------------------------+---------------+---------+----------------+
| id | name | enabled | email |
+----------------------------------+---------------+---------+----------------+
| fdc9314674ae49e797b4258226c20009 | admin | True | |
| 7299f9f6d5be4c9dbb615869dbeb846b | cmdb | True | |
| e237c9cb2bca4c25a570e4c21f0af906 | user123456 | True | <email address hidden> |
| 7c0ddd1e33384da683588ad49e1afdd1 | user1234567 | True | |
| e3477d7cf02749b1b505d2fa5b5d6ede | glance | True | |
| edf47979c2ce4931b1b9a2d1f9859931 | neutron | True | |
| a99d5a80c19a465ca7c46712152fb410 | neutron-admin | True | |
| bb9394c596bc44cc82fb87da86c5f1de | neutron-nova | True | |
| 490501e90c1047399de72403f088b2af | nova | True | |
| 8f9d271dd37c458e94ba6f9e6e20a786 | nova-notify | True | |
+----------------------------------+---------------+---------+----------------+

if i change /etc/keystone/keystone.conf
like this

public_endpoint = https://keystone.lan.yygamedev.com:5000
admin_endpoint = https://keystone.lan.yygamedev.com:35357

to
public_endpoint = https://keystone.lan.yygamedev.com:5000/v2.0/
admin_endpoint = https://keystone.lan.yygamedev.com:35357/v2.0/

error is, more or less, the same. in both cases keystoneclient works fine

[Steve Martinelli (stevemar)]

I'm not sure whats going on here, but this call is the root of the problem:

DEBUG: keystoneclient.session REQ: curl -g -i --insecure -X GET https://keystone.lan.example.com:35357/v2.0 -H "Accept: application/json" -H "User-Agent: python-openstackclient"

DEBUG: keystoneclient.session RESP: [404] content-length: 90 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.2.22 (Ubuntu) connection: Keep-Alive date: Fri, 11 Sep 2015 06:16:45 GMT content-type: application/json x-distribution: Ubuntu

RESP BODY: {"error": {"message": "Could not find version, v2.0.", "code": 404, "title": "Not Found"}}

Apparently, this URL can't be hit: https://keystone.lan.example.com:35357/v2.0 ?
Is this because you have it as .example.com instead of .yygamedev.com?

[gustavo panizzo (gfa)]

> Apparently, this URL can't be hit: https://keystone.lan.example.com:35357/v2.0 ?

that's right, but it does not stop keystoneclient from working

$ curl https://keystone.lan.example.com:35357/v2.0
{"error": {"message": "Could not find version, v2.0.", "code": 404, "title": "Not Found"}}%

$ curl https://keystone.lan.yygamedev.com:5000/v2.0
{"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "https://keystone.lan.yygamedev.com:5000/v2.0/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/content/", "type": "text/html", "rel": "describedby"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/identity-dev-guide-2.0.pdf", "type": "application/pdf", "rel": "describedby"}]}}

> Is this because you have it as .example.com instead of .yygamedev.com?

no, all the configs are yygamedev.com I just miss to anonymize them in the bug report

[Steve Martinelli (stevemar)]

this seems like a configuration error; check openstackclient docs: http://docs.openstack.org/developer/python-openstackclient/authentication.html#authenticating-using-identity-server-api-v3

and various sections of the install guide http://docs.openstack.org/liberty/install-guide-ubuntu/keystone-openrc.html

feel free to add your catalog of services and what version of openstack you're using

[gustavo panizzo (gfa)]

my keystone servers are icehouse, I think it happens with kilo too but I don't have access to that environment anymore

$ keystone endpoint-list
+----------------------------------+--------------+------------------------------------------------------+-----------------------------------------------------
+-+------------------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl
+| adminurl | service_id |
+----------------------------------+--------------+------------------------------------------------------+-----------------------------------------------------
+-+------------------------------------------------------+----------------------------------+
| 10e0219eb2b34dbc8fea5ca661be1006 | zh-office-01 | https://nova.lan.yygamedev.com:8774/v2/%(tenant_id)s | https://nova.lan.yygamedev.com:8774/v2/%(tenant_id)s
+| https://nova.lan.yygamedev.com:8774/v2/%(tenant_id)s | af1d09c400d34beda4abbb7c57f550f5 |
| 6781367b31964a588f7e569bf65f87d1 | zh-office-01 | https://glance.lan.yygamedev.com:9292/ | https://glance.lan.yygamedev.com:9292/
+| https://glance.lan.yygamedev.com:9292/ | 914f0d37e9244600b27e026948b6534a |
| bb570d60fec24b8ebb0f1aae8f9e1140 | zh-office-01 | https://keystone.lan.yygamedev.com:5000/v2.0 | https://keystone.lan.yygamedev.com:5000/v2.0
+| https://keystone.lan.yygamedev.com:35357/v2.0 | fb08e0a35a6d470aad4419ba2a26aff4 |
| ed52186aedcc4e3d9d00670b660665c0 | zh-office-01 | https://neutron.lan.yygamedev.com:9696 | https://neutron.lan.yygamedev.com:9696
+| https://neutron.lan.yygamedev.com:9696 | 635ba360792946e49e9ff636727ea670 |
+----------------------------------+--------------+------------------------------------------------------+-----------------------------------------------------
+-+------------------------------------------------------+----------------------------------+

$ cat openrc.admin
unset OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT OS_TENANT_NAME OS_USERNAME OS_PASSWORD OS_AUTH_URL
export OS_USERNAME=admin
export OS_PASSWORD=xxxxxxxxx
export OS_TENANT_NAME=admin
export OS_AUTH_URL=https://keystone.lan.yygamedev.com:35357/v2.0
export ADMIN_TENANT_ID=32669baf5beb4eed812212dae7ee0182
export OS_REGION_NAME=zh-office-01
export OS_NO_CACHE=1

I can reproduce the problem when I put keystone behind apache