Wrong OS_AUTH_URL makes keystone operations fail

[Kyrylo Romanenko]

Every keystone command fails with error: Authorization Failed: The resource could not be found. (HTTP 404)

Example:
root@node-1:~# keystone tenant-create --name demo
/usr/lib/python2.7/dist-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py:376: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
Authorization Failed: The resource could not be found. (HTTP 404)

Steps to reproduce:
1. Deploy MOS cluster.
2. SSH to Controller node.
3. source openrc.
4. type "keystone tenant-create --name demo", or "keystone user-list" etc

Contents of openrc:

root@node-1:~# cat openrc
#!/bin/sh
export LC_ALL=C
export OS_NO_CACHE='true'
export OS_TENANT_NAME='admin'
export OS_PROJECT_NAME='admin'
export OS_USERNAME='admin'
export OS_PASSWORD='admin'
export OS_AUTH_URL='http://192.168.0.2:5000/'
export OS_DEFAULT_DOMAIN='default'
export OS_AUTH_STRATEGY='keystone'
export OS_REGION_NAME='RegionOne'
export CINDER_ENDPOINT_TYPE='internalURL'
export GLANCE_ENDPOINT_TYPE='internalURL'
export KEYSTONE_ENDPOINT_TYPE='internalURL'
export NOVA_ENDPOINT_TYPE='internalURL'
export NEUTRON_ENDPOINT_TYPE='internalURL'
export OS_ENDPOINT_TYPE='internalURL'
export MURANO_REPO_URL='http://storage.apps.openstack.org/'

See that export OS_AUTH_URL='http://192.168.0.2:5000/',
here suffix v2.0 is lost.
To fix this it should be:
export OS_AUTH_URL='http://192.168.0.2:5000/v2.0'

Cluster configuration:
Controller+CephOSD
Controller+CephOSD
Compute
Mongo
Ironic

Neutron VLAN network, Ceph for all storages.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[Timur Nurlygayanov (tnurlygayanov)]

Hi team,

we are going to investigate why this issue was not detected by our automated tests, it was fund during manual acceptance of RC1.

It looks like it is regression issue which will affect Keystone CLI (other services work fine, WebUI works fine as well).

And we have the workaround for the issue (manually fix openrc file)

Workaround:
Execute on all controller nodes:

sed -i 's/5000\//5000\/v2.0\//' /root/openrc
source /root/openrc

[Roman Podoliaka (rpodolyaka)]

IMO, this should go to MU1 as a High bug:

keystoneclient has been deprecated for a while in favor of openstackclient. The latter actually works, as well as any other python-*client except for the Keystone one.

There is a workaround, described by Timur above. And, FWIW, credentials in openrc file is mostly for convenience - it's not required for using of OpenStack APIs.

[Boris Bobrov (bbobrov)]

I am setting this as "won't fix". `keystone` cli utility is deprecated and should not be used; it will probably be removed in 9.0. The issue doesn't affect work the cloud. There is a very easy workaround for it.

[Timur Nurlygayanov (tnurlygayanov)]

The issue reproduced not only for keystone CLI client.

[Alexey Galkin (agalkin)]

Swift CLI client not work with current openrc.

root@node-3:~# source openrc

root@node-3:~# cat openrc
#!/bin/sh
export LC_ALL=C
export OS_NO_CACHE='true'
export OS_TENANT_NAME='admin'
export OS_PROJECT_NAME='admin'
export OS_USERNAME='admin'
export OS_PASSWORD='admin'
export OS_AUTH_URL='http://192.168.0.2:5000/'
export OS_DEFAULT_DOMAIN='default'
export OS_AUTH_STRATEGY='keystone'
export OS_REGION_NAME='RegionOne'
export CINDER_ENDPOINT_TYPE='internalURL'
export GLANCE_ENDPOINT_TYPE='internalURL'
export KEYSTONE_ENDPOINT_TYPE='internalURL'
export NOVA_ENDPOINT_TYPE='internalURL'
export NEUTRON_ENDPOINT_TYPE='internalURL'
export OS_ENDPOINT_TYPE='internalURL'
export MURANO_REPO_URL='http://storage.apps.openstack.org/'

root@node-3:~# swift list
Authorization Failure. Authorization Failed: The resource could not be found. (HTTP 404)

root@node-3:~# vim openrc

root@node-3:~# source openrc

root@node-3:~# cat openrc
#!/bin/sh
export LC_ALL=C
export OS_NO_CACHE='true'
export OS_TENANT_NAME='admin'
export OS_PROJECT_NAME='admin'
export OS_USERNAME='admin'
export OS_PASSWORD='admin'
export OS_AUTH_URL='http://192.168.0.2:5000/v2.0'
export OS_DEFAULT_DOMAIN='default'
export OS_AUTH_STRATEGY='keystone'
export OS_REGION_NAME='RegionOne'
export CINDER_ENDPOINT_TYPE='internalURL'
export GLANCE_ENDPOINT_TYPE='internalURL'
export KEYSTONE_ENDPOINT_TYPE='internalURL'
export NOVA_ENDPOINT_TYPE='internalURL'
export NEUTRON_ENDPOINT_TYPE='internalURL'
export OS_ENDPOINT_TYPE='internalURL'
export MURANO_REPO_URL='http://storage.apps.openstack.org/'

root@node-3:~# swift list
root@node-3:~#

[Robert Duncan (rduncan-t)]

I am a customer using keystone v3 in Juno (MOS 6.1)
As far as I can tell keystone is supposed to have an un-versioned endpoint as deployed by fuel 8.0
keystone presents a http 300 'multiple choices' so the python keystone client is supposed to display a depreciation warning and work and the openstack client is supposed to work also.
moreover - clients are supposed to be moving away from a hard coded keystone url
and lastly - we can also export both of the endpoints with 2 slightly different variables -

export OS_AUTH_URL='http://keystone:35357/v2.0'
export OS_URL='http://keystone:35357/v3'

but this shouldn't be necessary.

From Docs -

If you were to perform a GET against either of these endpoints, you would be greeted by an HTTP/1.1 300 Multiple Choices response, which newer Keystone clients can use to automatically detect available API versions.

$ curl -i http://identity:35357/
HTTP/1.1 300 Multiple Choices
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 755
Date: Tue, 10 Jun 2014 14:22:26 GMT

{"versions": {"values": [ ... ]}}
With unversioned identity endpoints in the service catalog, you should be able to authenticate with keystoneclient successfully.

[Robert Duncan (rduncan-t)]

by the way ceph radosgw has a hardcoded keystone url of /v2.0

[Alex Schultz (alex-schultz)]

Should we just be adding the AUTH_VERSION (see Bug 1563057) rather than appending v2.0 on the end of the url? This way it can still be configured to use 2 or 3 without having to manage the url. My understanding is that the versionless url should be used going forward and if you need to specify 2 or 3 then do so via the AUTH_VERSION variable

[Alexey Deryugin (velovec)]

According to python-swiftclient code base [0], swift client uses old auth mechanism, which is deprecated [1].
So Swift CLI client doesn't support new features like versionless AUTH_URL, or at least AUTH_VERSION parameter.

Since OpenStack CLI client support operations on object storage, it should be used instead of Swift CLI client.

[0] https://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L518-L533
[1] https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/v2_0/client.py#L151-L156