keystone v3 :: nova and glance cli failing with error after configuring v3.

[Ritam Gangopadhyay]

build 3 kilo setup - multi node HA

root@nodeg34:~# nova list
ERROR (BadRequest): Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-b887a077-111b-4e3a-8be8-c933aee1cc39)
root@nodeg34:~# nova service-list
ERROR (BadRequest): Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-d0dcceef-8322-4065-a69e-f3bd242abe05)
root@nodeg34:~# glance image-list
Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-bce4655b-6f51-4cba-a400-a1a3fe964e40)
root@nodeg34:~#

Setup:-

host1 = 'root@10.204.221.24'
host2 = 'root@10.204.221.27'
host3 = 'root@10.204.221.28'
host4 = 'root@10.204.221.25'
host5 = 'root@10.204.221.26'

env.roledefs = {

    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1, host2, host3],
    'openstack':[host1, host2, host3],
    'control':[host1, host2, host3],
    'compute': [host4, host5],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],

}
env.hostnames ={
    'all': ['nodeg34', 'nodec48', 'nodec49', 'nodec51', 'nodec63']
}

env.ha = {
    'internal_vip' : '192.168.100.10',
    'external_vip' : '10.204.221.17',
}

[Ritam Gangopadhyay (ritam)]

root@nodeg34:~# nova --debug list
DEBUG (session:195) REQ: curl -g -i -X GET http://192.168.100.10:5000/v3/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG (retry:155) Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0)
INFO (connectionpool:188) Starting new HTTP connection (1): 192.168.100.10
DEBUG (connectionpool:362) "GET /v3/ HTTP/1.1" 200 253
DEBUG (session:223) RESP: [200] content-length: 253 vary: X-Auth-Token x-distribution: Ubuntu date: Thu, 28 Jul 2016 09:59:22 GMT content-type: application/json x-openstack-request-id: req-f2db4d4e-11b7-4366-a73d-80695c454868
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://192.168.100.10:5000/v3/", "rel": "self"}]}}

DEBUG (v3:125) Making authentication request to http://192.168.100.10:5000/v3/auth/tokens
DEBUG (retry:155) Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0)
DEBUG (connectionpool:362) "POST /v3/auth/tokens HTTP/1.1" 400 236
DEBUG (session:388) Request returned failure status: 400
DEBUG (shell:914) Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-cb4955f8-7e1c-4ba1-b65b-216b3fba4665)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 911, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 838, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/shell.py", line 1377, in do_list
    sort_dirs=sort_dirs)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 617, in list
    return self._list("/servers%s%s" % (detail, query_string), "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 164, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 200, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 89, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 307, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 589, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/base.py", line 114, in get_headers
    token = self.get_token(session)
  File...

[Deepinder Setia (dsetia)]

What errors glance and neutron returning? Have you tried “openstack” client? I know keystone client doesn’t handle V3 well (keystone tenant-list would fail, for example) but not sure about glance/neutron.

I have also seen assigning “admin” user “admin” role in default domain (in addition to admin tenant) also helps.

—
Deepinder

OOO 8/31

On 7/29/16, 12:31 AM, "Ritam Gangopadhyay" <email address hidden> wrote:

>Thanks,
>
> It works with following set of parameters set in openstackrc file
>
>export OS_USERNAME=admin
>export OS_PASSWORD=contrail123
>export OS_AUTH_URL=http://192.168.100.10:5000/v3/
>export OS_NO_CACHE=1
>export OS_DEFAULT_DOMAIN=default
>export OS_PROJECT_NAME=admin
>export OS_IDENTITY_API_VERSION=3
>
>
>but glance and neutron api's are still running into authentication errors
>
>
>root@nodeg34:~# glance image-list
>You must provide a project_id or project_name (with project_domain_name or project_domain_id) via --os-project-id (env[OS_PROJECT_ID]) --os-project-name (env[OS_PROJECT_NAME]), --os-project-domain-id (env[OS_PROJECT_DOMAIN_ID]) --os-project-domain-name (env[OS_PROJECT_DOMAIN_NAME])
>root@nodeg34:~# glance --os-project-id admin image-list
>The request you have made requires authentication. (HTTP 401) (Request-ID: req-5e17092e-979a-461d-87ef-4ddabc2ff2b7)
>root@nodeg34:~# glance --debug --os-project-id admin image-list
>Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/glanceclient/shell.py", line 668, in main
> OpenStackImagesShell().main(argv)
> File "/usr/local/lib/python2.7/dist-packages/glanceclient/shell.py", line 552, in main
> client = self._get_versioned_client('2', args)
> File "/usr/local/lib/python2.7/dist-packages/glanceclient/shell.py", line 447, in _get_versioned_client
> region_name=args.os_region_name)
> File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 679, in get_endpoint
> return auth.get_endpoint(self, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 329, in get_endpoint
> service_catalog = self.get_access(session).service_catalog
> File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 254, in get_access
> self.auth_ref = self.get_auth_ref(session)
> File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py", line 191, in get_auth_ref
> authenticated=False, log=False, **rkwargs)
> File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 519, in post
> return self.request(url, 'POST', **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/wrapt/wrappers.py", line 561, in __call__
> args, kwargs)
> File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
> return wrapped(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 419, in request
> raise exceptions.from_response(resp, method, url)
>Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-2d4f1526-1f74-467e-9fbc-ad481e1...

[Deepinder Setia (dsetia)]

Should be fixed by:

https://review.opencontrail.org/#/c/23013/ and
https://review.opencontrail.org/#/c/23058/

[Ritam Gangopadhyay (ritam)]

Verified on build 25 mitaka

[Jeba Paulaiyan (jebap)]

Please make sure if equivalent of this is addressed in 3.2 and 4.0 branches

https://review.opencontrail.org/#/c/23013/ and
https://review.opencontrail.org/#/c/23058/