Keystone V3 authentication return BadRequest: Malformed request url

[Jin Liu]

When using keystone V3 authentication for cinder and nova (see comment #3), I got error "BadRequest: Malformed request url (HTTP 400)".
I am testing on Juno release, my keystone v3 env is like this,

export OS_USERNAME="admin"
export OS_PASSWORD="password"
export OS_DOMAIN_NAME=default
export OS_AUTH_URL="http://$MY_HOST:35357/v3"
export OS_IDENTITY_API_VERSION=3

My endpoint of cinder public URL is like http://**.**.**.**:8776/v1/cbe4b1d87fbb4318be379a79a570b7ec (I hided the real IP)
When run command "openstack --debug volume list" or "openstack --debug volume create --size 1 jin", I got this BadRequest error. From debug info, this error comes from cinder server. I added log in cinder/api/openstack/wsgi.py function _process_stack(), found the context.project_id is None while project_id has a value, here return the error.

if (context and project_id and (project_id != context.project_id)):
            msg = _("Malformed request url")
            return Fault(webob.exc.HTTPBadRequest(explanation=msg))

I compared with another keystone V2 authentication server, the context.project_id is same as project_id. Maybe this is difference, in v2 server the REQ has one more Project-id like "curl -i -H "X-Auth-Project-Id: admin".
I found the cinder.context maybe come from cinder/api/middleware/auth.py, the project_id in cinder.context may not be assigned a value in keystone v3 authentication scenario.

ERROR log is as below:

REQ: curl -i http://**.**.**.**:8776/v1/cbe4b1d87fbb4318be379a79a570b7ec/volumes/detail -X GET -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: e883e05a887144d4ae70151c976ce666"

INFO: requests.packages.urllib3.connectionpool Starting new HTTP connection (1): **.**.**.**
DEBUG: requests.packages.urllib3.connectionpool "GET /v1/cbe4b1d87fbb4318be379a79a570b7ec/volumes/detail HTTP/1.1" 400 65
DEBUG: cinderclient.client RESP: [400] {'date': 'Thu, 09 Apr 2015 00:35:30 GMT', 'content-length': '65', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-39a96150-b9ab-4753-8b02-d5730492b288', 'x-openstack-request-id': 'req-39a96150-b9ab-4753-8b02-d5730492b288'}
RESP BODY: {"badRequest": {"message": "Malformed request url", "code": 400}}

ERROR: openstack Malformed request url (HTTP 400) (Request-ID: req-39a96150-b9ab-4753-8b02-d5730492b288)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 280, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 91, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/openstackclient/volume/v1/volume.py", line 255, in take_action
    data = volume_client.volumes.list(search_opts=search_opts)
  File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 220, in list
    "volumes")
  File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 70, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 302, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 269, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 252, in request
    raise exceptions.from_response(resp, body)
BadRequest: Malformed request url (HTTP 400) (Request-ID: req-39a96150-b9ab-4753-8b02-d5730492b288)

[Jin Liu (jin-t)]

Add full log before REQ,

 # openstack --debug volume list
DEBUG: openstackclient.shell compute API version 2, cmd group openstack.compute.v2
DEBUG: openstackclient.shell network API version 2, cmd group openstack.network.v2
DEBUG: openstackclient.shell image API version 1, cmd group openstack.image.v1
DEBUG: openstackclient.shell volume API version 1, cmd group openstack.volume.v1
DEBUG: openstackclient.shell identity API version 3, cmd group openstack.identity.v3
DEBUG: openstackclient.shell object_store API version 1, cmd group openstack.object_store.v1
DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('csv = cliff.formatters.commaseparated:CSVLister')
DEBUG: openstackclient.shell prepare_to_run_command ListVolume
DEBUG: openstackclient.shell validating authentication options
DEBUG: openstackclient.identity.client Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
DEBUG: openstackclient.identity.client Using password auth
DEBUG: keystoneclient.auth.identity.v3 Making authentication request to http://**.**.**.**:35357/v3/auth/tokens
INFO: requests.packages.urllib3.connectionpool Starting new HTTP connection (1): **.**.**.**
DEBUG: requests.packages.urllib3.connectionpool "POST /v3/auth/tokens HTTP/1.1" 201 4434
DEBUG: openstackclient.volume.v1.volume.ListVolume take_action(Namespace(all_projects=False, columns=[], formatter='table', long=False, max_width=0, name=None, quote_mode='nonnumeric', status=None))
DEBUG: openstackclient.volume.client Instantiating volume client: <class 'cinderclient.v1.client.Client'>
DEBUG: cinderclient.client
REQ: curl -i http://**.**.**.**:8776/v1/cbe4b1d87fbb4318be379a79a570b7ec/volumes/detail -X GET -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: 53a14c48b0db4023a9dc454dfc661e70"

[Jin Liu (jin-t)]

When delete the piece code of project_id checking in wsgi.py, my "openstack volume create" and "openstack volume list" all completed successfully. I suggest we may need handle both keystone v2 and v3 authentication well. Here is the print of request.environ in wsgi.py for reference.

HTTP_X_TENANT_NAME': None
'HTTP_X_ROLE': u'admin'
'HTTP_X_USER_ID': u'e3b7f7511f234afa926d8b905478ff85'
'HTTP_X_AUTH_TOKEN': '9902e483ac944b99836cd9592dbe0cdb'
'HTTP_X_DOMAIN_NAME': u'Default'
'HTTP_X_DOMAIN_ID': u'default'
'HTTP_X_PROJECT_DOMAIN_ID': None
'HTTP_X_TENANT_ID': None
'HTTP_X_PROJECT_DOMAIN_NAME': None
'HTTP_X_USER_DOMAIN_NAME': u'Default'
'HTTP_X_TENANT': None
'HTTP_X_USER': u'admin'
'HTTP_X_USER_DOMAIN_ID': u'default'
'HTTP_X_PROJECT_NAME': None
'HTTP_X_PROJECT_ID': None
'HTTP_X_USER_NAME': u'admin'

[Jin Liu (jin-t)]

When run "openstack server list", I got similar ERROR. There's also project_id check in nova/api/openstack/wsgi.py function _process_stack().

Error log is as below,

# openstack --debug server list
DEBUG: openstackclient.shell compute API version 2, cmd group openstack.compute.v2
DEBUG: openstackclient.shell network API version 2, cmd group openstack.network.v2
DEBUG: openstackclient.shell image API version 1, cmd group openstack.image.v1
DEBUG: openstackclient.shell volume API version 1, cmd group openstack.volume.v1
DEBUG: openstackclient.shell identity API version 3, cmd group openstack.identity.v3
DEBUG: openstackclient.shell object_store API version 1, cmd group openstack.object_store.v1
DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('csv = cliff.formatters.commaseparated:CSVLister')
DEBUG: openstackclient.shell prepare_to_run_command ListServer
DEBUG: openstackclient.shell validating authentication options
DEBUG: openstackclient.identity.client Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
DEBUG: openstackclient.identity.client Using password auth
DEBUG: keystoneclient.auth.identity.v3 Making authentication request to http://**.**.**.**:35357/v3/auth/tokens
INFO: requests.packages.urllib3.connectionpool Starting new HTTP connection (1): **.**.**.**
DEBUG: requests.packages.urllib3.connectionpool "POST /v3/auth/tokens HTTP/1.1" 201 4444
DEBUG: openstackclient.compute.v2.server.ListServer take_action(Namespace(all_projects=False, columns=[], flavor=None, formatter='table', host=None, image=None, instance_name=None, ip=None, ip6=None, long=False, max_width=0, name=None, quote_mode='nonnumeric', reservation_id=None, status=None))
DEBUG: openstackclient.compute.client Instantiating compute client: <class 'novaclient.v1_1.client.Client'>
DEBUG: openstackclient.compute.v2.server.ListServer search options: {'instance_name': None, 'status': None, 'host': None, 'ip6': None, 'name': None, 'ip': None, 'flavor': None, 'reservation_id': None, 'image': None, 'all_tenants': False}
REQ: curl -i 'http://**.**.**.**:8774/v2/b5167e1a791d4cd8af0855603f73b9a0/servers/detail' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}03aa4227ca578761f60664e623f490be33a79ff9"
RESP: [400] {'date': 'Fri, 10 Apr 2015 17:09:20 GMT', 'content-length': '160', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-c4ab0444-21f0-49a4-8223-dce1b9d22a1e'}
RESP BODY: null

ERROR: openstack Malformed request URL: URL's project_id 'b5167e1a791d4cd8af0855603f73b9a0' doesn't match Context's project_id 'None' (HTTP 400) (Request-ID: req-c4ab0444-21f0-49a4-8223-dce1b9d22a1e)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 280, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 91, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 579, in take_action
    data = c...

[Jin Liu (jin-t)]

Relate error in Nova flavor, caused by None project_id

# openstack flavor list
ERROR: openstack The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400) (Request-ID: req-b129987e-2902-421d-b7e4-e6d44dd3f669)

Part of the error log:

2015-04-10 19:42:14.629 504 TRACE nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/common.py", line 504, in _get_project_id
2015-04-10 19:42:14.629 504 TRACE nova.api.openstack.wsgi if project_id in request.url:
2015-04-10 19:42:14.629 504 TRACE nova.api.openstack.wsgi TypeError: 'in <string>' requires string as left operand, not NoneType

[Jin Liu (jin-t)]

I have a workaround now, if I don't put "export OS_DOMAIN_NAME=default" in ENV, instead add domain in CLI like this,

# openstack --debug --os-project-name admin --os-project-domain-name default --os-user-domain-name default server list

# openstack --debug --os-project-name admin --os-project-domain-name default --os-user-domain-name default volume list

I got all commands working and finally boot VM successfully in keystone V3 authentication.

[Jin Liu (jin-t)]

Some updates, we made keystone v3 working fine either using cli or horizon.
We figured out what parameters required in openstackclient command line. Using example in this ticket,
# openstack volume list
# openstack server list
Actually these two commands need OS_PROJECT_NAME env or --os-project-name in command line. Export OS_DOMAIN_NAME will not help. Basically OS_PROJECT_NAME is needed for most services(nova/cinder/neutron etc) except Identity. Identity requires OS_DOMAIN_NAME.
You may think export OS_PROJECT_NAME and OS_DOMAIN_NAME together. But openstackclient will complain when you do both,
"ERROR: openstack Authentication cannot be scoped to multiple targets. Pick one of: project, domain or trust".

You can find extra info about policy.json from this article https://www.mirantis.com/blog/manage-openstack-projects-using-domains-havana/

[Jin Liu (jin-t)]

Code change is not needed on Cinder/Nova server, just some conf to use keystone v3 authentication.

[David Cheperdak (djbchepe)]

Hello Jin,

Is this still a bug in OpenStack Compute Nova that needs a fix?

Thank you,