nova attach volume failed when using keystone v3 API token

[LiShaokai]

reproduce:

1 . Get tokens of V3; POST http://1.2.3.4:5000/v3/auth/tokens

2. Use that token to attach a volume for a instance with the following request:
POST http://10.9.1.147:8774/v2/e7a9800dba03489ca2716ab03226a277/servers/9c1f855d-20d5-47d7-984c-01b4f84203f5/os-volume_attachments

{
  "volumeAttachment":{
        "device":null,
        "volumeId":"6ed17ada-0d94-4dc1-86c1-690035e4b874"
  }
}

An Error occurred : 500 internal ERROR.
IF we use keystone v2 token, attach volume is ok.

V3 service catalog:

 [{u'endpoints': [{u'id': u'6c205a937fc94cd89ef6d25804631742',
                  u'interface': u'internal',
                  u'legacy_endpoint_id': u'52d0fe4693094e29a1f3737aa6a74024',
                  u'region': u'RegionOne',
                  u'url': u'http://10.9.1.147:8776/v1/e7a9800dba03489ca2716ab03226a277'},
                  {u'id': u'c289637b673643b69565a04ed1561a4c',
                  u'interface': u'public',
                  u'legacy_endpoint_id': u'52d0fe4693094e29a1f3737aa6a74024',
                  u'region': u'RegionOne',
                  u'url': u'http://10.9.1.147:8776/v1/e7a9800dba03489ca2716ab03226a277'},
                  {u'id': u'f23ede3238974e0685b5e478971a566d',
                   u'interface': u'admin',
                   u'legacy_endpoint_id': u'52d0fe4693094e29a1f3737aa6a74024',
                   u'region': u'RegionOne',
                  u'url': u'http://10.9.1.147:8776/v1/e7a9800dba03489ca2716ab03226a277'}],
                  u'id': u'9206e300379a448592df585587c68a40',
                  u'type': u'volume'}]

V2 service catalog:

 [{u'endpoints': [{u'adminURL': u'http://10.9.1.147:8776/v1/e7a9800dba03489ca27',
               u'id': u'6c205a937fc94cd89ef6d25804631742',
               u'internalURL': u'http://10.9.1.147:8776/v1/e7a9800dba03486ab03226a277',
               u'publicURL': u'http://10.9.1.147:8776/v1/e7a9800dba03489c6ab03226a277',
               u'region': u'RegionOne'}]

So below code is the break in keystone V3: in cinderclient/service_catalog.py, function url_for

in LINE 83, the length of matching_endpoints is 3, so that An AmbiguousEndpoints Exception are thrown.
[code]

     80 if not matching_endpoints:
     81 raise cinderclient.exceptions.EndpointNotFound()
     82 elif len(matching_endpoints) > 1:
     83 raise cinderclient.exceptions.AmbiguousEndpoints(
     84 endpoints=matching_endpoints)
     85 else:
     86 return matching_endpoints[0][endpoint_type]

[/code]

[Brant Knudson (blk-u)]

A note for whoever winds up fixing this - keystoneclient provides a keystoneclient.service_client.ServiceCatalog class[1] that provides support for both the v2 and v3 service catalogs.

It looks like cinderclient has its own version of this... maybe cinderclient should use the one from python-keystoneclient or copy keystoneclient's again.

[1] http://git.openstack.org/cgit/openstack/python-keystoneclient/tree/keystoneclient/service_catalog.py?id=ce6c54de91f7c129cea11a7648f8668b2f24bca9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/64792

[lirenke (lvhancy)]

look this:
https://bugs.launchpad.net/python-novaclient/+bug/1154809
same bug?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/74598

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/76109

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/76109
Committed: https://git.openstack.org/cgit/openstack/python-cinderclient/commit/?id=1b0ea8768c9e1368e978a45b1bba125cd81e9b59
Submitter: Jenkins
Branch: master

commit 1b0ea8768c9e1368e978a45b1bba125cd81e9b59
Author: Shao Kai Li <email address hidden>
Date: Wed Feb 19 00:34:30 2014 -0500

    Import access module from keystoneclient to handle V3 endpoints

    * service_catalog.py will be used only by nova(cinder.py), it will be removed if nova
    uses access instead of service_catalog. Then service_catalog.py and
    test_service_catalog.py will be removed from cinderclient if necessary.

    * Some unit tests are modified.

    * Because of JSON format's modification, functions that process cinder
    credentials and cinder endpoints are changed.

    * Add dependency for keystoneclient in requirements.txt.

    Change-Id: Icf7badfdddcf5f55536d95db7242aff58aa34b6e
    Closes-Bug: #1263876
    bp: service-catalog

[Andriy Yurchuk (ch00k)]

Why is this only fixed in python-cinderclient? The bug states that a direct API request was sent to reproduce this.
I am having the exact same behavior - when I send an API request to attach a volume I get AmbiguousEndpoints exception. I think a fix must also be provided for the server-side code, not only client-side.

[Andriy Yurchuk (ch00k)]

Looks like there is already some work being done in this area: https://review.openstack.org/#/c/77524/

[Mike Spreitzer (mike-spreitzer)]

This reminds me of bug 1289135

[Jordan Pittier (jordan-pittier)]

I am also hitting this.

cinder_api_insecure = True
cinder_catalog_info = volume:cinder:internalURL

New info : this works :
auth_uri = https://identity.cw-labs.net/v2.0
auth_host = identity.usr.lab0.aub.cw-labs.net
auth_port = 5000

This doesn't work (raise AmbiguousEndpoints)
auth_uri = https://identity.cw-labs.net/v2.0
auth_host = identity.usr.lab0.aub.cw-labs.net
auth_port = 35357