Attaching volume fails if keystone has multiple endpoints of Cinder

[Rohit Karajgi]

Cinder fails to attach a volume to an instance if multiple endpoints of Cinder are configured in Keystone and
the endpoints are unique to a region.

Pre-config: Create a Cinder endpoint for two different regions (say, RegionOne and RegionTwo) in Keystone

Create a volume and try to attach it to an instance fails with the following error:

AmbiguousEndpoints: [{u'adminURL': u'http://10.2.3.102:8776/v1/4f272085adb34eef8a8b6e61e15f5009', u'region': u'RegionOne', u'internalURL': u'http://10.2.3.102:8776/v1/4f272085adb34eef8a8b6e61e15f5009', 'serviceName': u'cinder', u'id': u'1ebcb30a728948a29e594cdabe1d3ca5', u'publicURL': u'http://10.2.3.102:8776/v1/4f272085adb34eef8a8b6e61e15f5009'}, {u'adminURL': u'http://10.2.3.102:8776/v1/4f272085adb34eef8a8b6e61e15f5009', u'region': u'RegionTwo', u'internalURL': u'http://10.2.3.102:8776/v1/4f272085adb34eef8a8b6e61e15f5009', 'serviceName': u'cinder', u'id': u'6f17a81430cb4409af0026d543d362f7', u'publicURL': u'http://10.2.3.102:8776/v1/4f272085adb34eef8a8b6e61e15f5009'}]

Nova API Log snip: http://paste.openstack.org/show/27572/

[Michael Still (mikal)]

This looks like a bug in cinder client. I think nova gets this fixed for free if cinderclient is tweaked?

[Yaguang Tang (heut2008)]

this is expected, you need to specify the REGION_NAME by add OS_REGION_NAME=RegionOne to you env.
all clients now cann't works if multi endpoints is returns after filtering by using credentials offerred by user .

see the logic

if not matching_endpoints:
 80 raise exceptions.EndpointNotFound()
 81 elif len(matching_endpoints) > 1:
 82 raise exceptions.AmbiguousEndpoints(message=matching_endpoints)
 83 else:
 84 return matching_endpoints[0][endpoint_type]

the issue is should we allow multi endpoints being used at the same time.

[Rohit Karajgi (rohitk)]

@heut2008: IMO, this is a valid bug, the cinderclient code does not utilize the OS_REGION_NAME environment variable even if it's set.

It seems none of the clients/API seem to utilize the os-region-name parameter when passed.

Using novaclient:--->

$ nova --os-username demo --os-password pass --os-tenant-name demo --os-auth-url http://10.2.3.102:5000/v2.0 --os-region-name RegionTwo volume-attach 1dc1a92e-9668-42cc-9d9e-b00074d2e1e3 3e12340c-dc3c-4e6b-a69d-0495508b3821 /dev/vdc
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-387766dd-7e41-48d7-aed9-f375427819ae)

n-api log shows the same error as pasted in the description

Using cinderclient to list endpoints for a specific region,
(the client does not honor the region (I think this is a separate bug)): --->

$ cinder --os-region-name RegionTwo endpoints
Could not find any suitable endpoint. Correct region?
ERROR:

Using keystone client to list endpoints specific to a region (lists all the endpoints): --->

$ keystone --os-region-name RegionTwo endpoint-list
+----------------------------------+-----------+---------------------------------------------------------+---------------------------------------------------------+---------------------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------------------+---------------------------------------------------------+---------------------------------------------------------+----------------------------------+
| 1a8e75a0b34f44768ee49780ff71e6e0 | RegionOne | http://10.2.3.102:8776/v1/$(tenant_id)s | http://10.2.3.102:8776/v1/$(tenant_id)s | http://10.2.3.102:8776/v1/$(tenant_id)s | caed87251c8f436ab3da6d40ca728dd3 |
| 2c21d0a2320f458a97789c51238c62e7 | RegionOne | http://10.2.3.102:9292 | http://10.2.3.102:9292 | http://10.2.3.102:9292 | e9714b9423d8462f8d9332bfe1d03b8f |
| 3d22976758a0419e94527686fd52a667 | RegionTwo | http://10.2.3.102:8776$(compute_port)s/v2/$(tenant_id)s | http://10.2.3.102:8776$(compute_port)s/v2/$(tenant_id)s | http://10.2.3.102:8776$(compute_port)s/v2/$(tenant_id)s | caed87251c8f436ab3da6d40ca728dd3 |
| 44d7ea8d003c453e80b0ffc710acd4a5 | RegionOne | http://10.2.3.102:8773/services/Cloud | http://10.2.3.102:8773/services/Cloud | http://10.2.3.102:8773/services/Admin | 5a4d70b8959448f089046bbe79016136 |
| 45bb2976c3274c8a86629ce9fbe3e478 | RegionOne | http://10.2.3.102:$(public_port)s/v2.0 | http://10.2.3.102:$(public_port)s/v2.0 | http://10.2.3.102:$(admin_port)s/v2.0 | 606ab90bff4a41d08bd3f1be14592014 |
| 970eca3c262c47fa988124af35320512 | Reg...

[Yaguang Tang (heut2008)]

Rohit:

are you using the latest code? I have tested that both python-novaclient and python-cinderclient can handle the --os-region-name
option. when specify the --os-region-name the client can get the endpoints to that region.

root@yaguang-ThinkPad-X230:~/working/openstack/python-novaclient# nova --os-region-name Regiontwo endpoints
+-------------+-------------------------------------------------------------+
| nova | Value |
+-------------+-------------------------------------------------------------+
| adminURL | http://localhost:8774/v1.1/f9cd6823327b4ce4b76c68270006fb19 |
| id | 9c1ac5fc96a4473ba159a0fb94b92b3b |
| internalURL | http://localhost:8774/v1.1/f9cd6823327b4ce4b76c68270006fb19 |
| publicURL | http://localhost:8774/v1.1/f9cd6823327b4ce4b76c68270006fb19 |
| region | RegionOne |
+-------------+-------------------------------------------------------------+
+-------------+----------------------------------+
| quantum | Value |
+-------------+----------------------------------+
| adminURL | http://127.0.0.1:9696 |
| id | c06bfb948bc846debc5ff44f876f3871 |
| internalURL | http://127.0.0.1:9696 |
| publicURL | http://127.0.0.1:9696 |
| region | RegionOne |
+-------------+----------------------------------+
+-------------+----------------------------------+
| glance | Value |
+-------------+----------------------------------+
| adminURL | http://localhost:9292/v1 |
| id | 7d95ee08e9974694b2193e9bc1259234 |
| internalURL | http://localhost:9292/v1 |
| publicURL | http://localhost:9292/v1 |
| region | RegionOne |
+-------------+----------------------------------+
+-------------+-----------------------------------------------------------+
| nova-volume | Value |
+-------------+-----------------------------------------------------------+
| adminURL | http://localhost:8776/v1/f9cd6823327b4ce4b76c68270006fb19 |
| id | 7450aa27aae74801a18e53dfcc549148 |
| internalURL | http://localhost:8776/v1/f9cd6823327b4ce4b76c68270006fb19 |
| publicURL | http://localhost:8776/v1/f9cd6823327b4ce4b76c68270006fb19 |
| region | RegionOne |
+-------------+-----------------------------------------------------------+
+-------------+--------------------------------------+
| ec2 | Value |
+-------------+--------------------------------------+
| adminURL | http://localhost:8773/services/Admin |
| id | 816b17f4be6c4d329ad804d9c2cef93e |
| internalURL | http://localhost:8773/services/Cloud |
| publicURL | http://localhost:8773/services/Cloud |
| region | RegionOne |
+-------------+--------------------------------------+
+-------------+----------------------------------+
| keystone ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/18048

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/18048
Committed: http://github.com/openstack/nova/commit/4db98faa1c55c6f369ec84ed478338141ad5e7d8
Submitter: Jenkins
Branch: master

commit 4db98faa1c55c6f369ec84ed478338141ad5e7d8
Author: Rohit Karajgi <email address hidden>
Date: Mon Jan 7 03:22:16 2013 -0800

    Adds a flag to allow configuring a region

    If multiple regions contain an endpoint, extract
    the correct endpoint URL from the os_region_name parameter
    in nova.conf

    Fixes LP Bug #1087735

    Change-Id: Id4e4f56f336cd2b092a173b3c8e695b173c76f2a