OpenStack Compute (nova)

Volume detach fails via OSAPI: AmbiguousEndpoints

Bug #1154809 reported by Adam Gandelman
52
This bug affects 10 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Critical
Vish Ishaya
OpenStack Compute (nova) 2013.1 "grizzly"
python-cinderclient
Invalid
High
Unassigned
python-keystoneclient
Invalid
Wishlist
Unassigned
python-novaclient
Fix Released
High
Unassigned
python-novaclient 2.21.0

Bug Description

Not sure if this is a cinderclient bug or nova. Attempting to detach a volume via the OSAPI ends with an AmbiguousEndpoints exception:

2013-03-13 17:30:40.314 ERROR nova.api.openstack [req-9dc2a448-c2ed-4db3-be2f-6e1f8971d463 f1a96ff6310042f7b7a9b5acbb634a43 e00a289cba9b45169f054067d7dd74e1] Caught error: AmbiguousEndpoints: [{u'url': u'http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1', u'region': u'RegionOne', u'legacy_endpoint_id': u'8012ff386ffa4955b3eab965a4826b0e', 'serviceName': None, u'interface': u'internal', u'id': u'3b41c544eb24440b89946ab4da3c2524'}, {u'url': u'http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1', u'region': u'RegionOne', u'legacy_endpoint_id': u'8012ff386ffa4955b3eab965a4826b0e', 'serviceName': None, u'interface': u'public', u'id': u'981469fb6b4f4928b1a59783bec445c4'}, {u'url': u'http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1', u'region': u'RegionOne', u'legacy_endpoint_id': u'8012ff386ffa4955b3eab965a4826b0e', 'serviceName': None, u'interface': u'admin', u'id': u'c578676fd1c6404c9a4fddd8daf531a4'}]
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack Traceback (most recent call last):
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return req.get_response(self.application)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack application, catch_exc_info=False)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 451, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return self.app(env, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 895, in __call__
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack content_type, body, accept)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 955, in _process_stack
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1035, in dispatch
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack return method(req=request, **action_args)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/volumes.py", line 452, in delete
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack volume_id=volume_id)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2332, in detach_volume
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack volume = self.volume_api.get(context, volume_id)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 191, in get
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack self._reraise_translated_volume_exception(volume_id)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 188, in get
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack item = cinderclient(context).volumes.get(volume_id)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 92, in cinderclient
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack endpoint_type=endpoint_type)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/cinderclient/service_catalog.py", line 85, in url_for
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack endpoints=matching_endpoints)
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'url': u'http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1', u'region': u'RegionOne', u'legacy_endpoint_id': u'8012ff386ffa4955b3eab965a4826b0e', 'serviceName': None, u'interface': u'internal', u'id': u'3b41c544eb24440b89946ab4da3c2524'}, {u'url': u'http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1', u'region': u'RegionOne', u'legacy_endpoint_id': u'8012ff386ffa4955b3eab965a4826b0e', 'serviceName': None, u'interface': u'public', u'id': u'981469fb6b4f4928b1a59783bec445c4'}, {u'url': u'http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1', u'region': u'RegionOne', u'legacy_endpoint_id': u'8012ff386ffa4955b3eab965a4826b0e', 'serviceName': None, u'interface': u'admin', u'id': u'c578676fd1c6404c9a4fddd8daf531a4'}]
2013-03-13 17:30:40.314 8945 TRACE nova.api.openstack

The configured keystone services and catalog looks like:

adam@test-01:~$ keystone catalog
Service: compute
+-------------+-----------------------------------------------------------------------------------+
| Property | Value |
+-------------+-----------------------------------------------------------------------------------+
| adminURL | http://test-05.os.magners.qa.lexington:8774/v1.1/e00a289cba9b45169f054067d7dd74e1 |
| id | 10b0009cd95a45d899eee12ce0bb1253 |
| internalURL | http://test-05.os.magners.qa.lexington:8774/v1.1/e00a289cba9b45169f054067d7dd74e1 |
| publicURL | http://test-05.os.magners.qa.lexington:8774/v1.1/e00a289cba9b45169f054067d7dd74e1 |
| region | RegionOne |
+-------------+-----------------------------------------------------------------------------------+
Service: s3
+-------------+---------------------------------------------+
| Property | Value |
+-------------+---------------------------------------------+
| adminURL | http://test-05.os.magners.qa.lexington:3333 |
| id | 519b3206ab584bbaa5e0166eaac2bea5 |
| internalURL | http://test-05.os.magners.qa.lexington:3333 |
| publicURL | http://test-05.os.magners.qa.lexington:3333 |
| region | RegionOne |
+-------------+---------------------------------------------+
Service: image
+-------------+------------------------------------------------+
| Property | Value |
+-------------+------------------------------------------------+
| adminURL | http://test-11.os.magners.qa.lexington:9292/v1 |
| id | 5c870d37fdba4cf58b43a53f291b8e59 |
| internalURL | http://test-11.os.magners.qa.lexington:9292/v1 |
| publicURL | http://test-11.os.magners.qa.lexington:9292/v1 |
| region | RegionOne |
+-------------+------------------------------------------------+
Service: volume
+-------------+---------------------------------------------------------------------------------+
| Property | Value |
+-------------+---------------------------------------------------------------------------------+
| adminURL | http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1 |
| id | 3b41c544eb24440b89946ab4da3c2524 |
| internalURL | http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1 |
| publicURL | http://test-07.os.magners.qa.lexington:8776/v1/e00a289cba9b45169f054067d7dd74e1 |
| region | RegionOne |
+-------------+---------------------------------------------------------------------------------+
Service: ec2
+-------------+------------------------------------------------------------+
| Property | Value |
+-------------+------------------------------------------------------------+
| adminURL | http://test-05.os.magners.qa.lexington:8773/services/Cloud |
| id | 5dc0b588166f496a9f7e38343cea582b |
| internalURL | http://test-05.os.magners.qa.lexington:8773/services/Cloud |
| publicURL | http://test-05.os.magners.qa.lexington:8773/services/Cloud |
| region | RegionOne |
+-------------+------------------------------------------------------------+
Service: identity
+-------------+---------------------------------------------------+
| Property | Value |
+-------------+---------------------------------------------------+
| adminURL | http://test-02.os.magners.qa.lexington:35357/v2.0 |
| id | 2469befa176047e9a2307b590bd49016 |
| internalURL | http://test-02.os.magners.qa.lexington:5000/v2.0 |
| publicURL | http://test-02.os.magners.qa.lexington:5000/v2.0 |
| region | RegionOne |
+-------------+---------------------------------------------------+
adam@test-01:~$ keystone service-list
+----------------------------------+----------+----------+----------------------------+
| id | name | type | description |
+----------------------------------+----------+----------+----------------------------+
| 1a803d81a1754abc9d1b4f9f806b8343 | cinder | volume | Cinder Volume Service |
| 30c39e60f05244bbb5a648e8d4f5b365 | ec2 | ec2 | EC2 Compatibility Layer |
| 57e567fc7af5404ba2a9e2643cd1b35a | keystone | identity | Keystone Identity Service |
| 7ad8e13d1cc24473a011bdc2fa6e227e | glance | image | Glance Image Service |
| 863bb1c4ce364858a9eaffa006ed9a44 | nova | compute | Nova Compute Service |
| a89a355e152e48e49a46ee592110d241 | s3 | s3 | S3 Compatible object-store |
+----------------------------------+----------+----------+----------------------------+
adam@test-01:~$

Revision history for this message
James Page (james-page) wrote :

I see the same when trying to attach volumes

Revision history for this message
Dolph Mathews (dolph) wrote :

Added keystone because it doesn't look like the service name attribute should be null here.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

I can't reproduce this in devstack, but this seems to be an issue with auth_token middleware trying to use v3 api and getting an incompatible catalog.

Can you try setting:

auth_version = v2.0

in the

[filter:authtoken]

section of /etc/nova/api-paste.ini

to see if that fixes it ?

Revision history for this message
Vish Ishaya (vishvananda) wrote :

It also occurs to me that v3 pki tokens will not work at all since the catalog will be in the wrong format in the token. We may need a fix in nova to do some translation of the service catalog to make that work.

Vish Ishaya (vishvananda)
Changed in nova:
importance: Undecided → Critical
milestone: none → grizzly-rc1
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Configuring v2.0 auth_version fixes the issue here, for both UUID + PKI token_formats. Thanks, Vish.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

This is an incompatiblilty between v2 and v3 service catalogs. This only happens with UUID tokens currently.

Nova gets the service catalog from auth_token middleware and then passes that to cinderclient. The problem breaks down as follows:

Using PKI:

novaclient still uses v2.0 so it gets a v2.0 pki token. The auth_token middleware simply extracts the catalog from the token and passes it along. The catalog is in v2.0 format

Using uuid:
novaclient uses v2.0 and gets a v2.0 uuid token. The auth_token middleware uses the most recent (v3.0) api to validate the token so it gives back a v3.0 catalog. This catalog is then passed in to cinderclient which blows up.

So clearly cinderclient need to support parsing the new catalog, but in the meantime we can get around this issue by forcing nova to use the v2.0 api Unfortunately if people start sending in v3 PKI tokens we will see a similar issue, so we really need either:

a) something that can translate v3-v2 catalogs in nova or b) an updated cinderclient that can understand v3 catalogs.

Changed in python-cinderclient:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/24386

Changed in nova:
assignee: nobody → Vish Ishaya (vishvananda)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/24386
Committed: http://github.com/openstack/nova/commit/4bf35503c21a2f54474634f0de4b19489384d56f
Submitter: Jenkins
Branch: master

commit 4bf35503c21a2f54474634f0de4b19489384d56f
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Mar 13 16:48:33 2013 -0700

    Force nova to use keystone v2.0 for auth_token

    This is a workaround for bug 1154809 until cinderclient supports
    v3 service catalogs or we have some workaround in place.

    Change-Id: I832ef1735579b240a367c051c44f97b1619f8d3d

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
Revision history for this message
Dolph Mathews (dolph) wrote :

Continuing vish's line of thought... keystoneclient is working towards exposing the catalog through a python API (c.service_catalog) rather than as a raw dict, so that other clients can utilize the catalog without having to understand it's HTTP format. Attached this bug to keystoneclient as I want to make sure that we have test coverage for the use case presented here.

affects: keystone → python-keystoneclient
Changed in python-keystoneclient:
status: New → Confirmed
importance: Undecided → Wishlist
Revision history for this message
Andrea Frittoli (andrea-frittoli) wrote :

My two cents: all python bindings should rely on python-keystoneclient for obtaining token and url from catalogue, rather than duplicating the logic to get a token and parse the catalogue, like cinder and nova client do. This will make it easier in future to support new version of the identity API.

Revision history for this message
lirenke (lvhancy) wrote :

"""
setting:
auth_version = v2.0
in the
[filter:authtoken]
section of /etc/nova/api-paste.ini
"""
does it just work in UUID token_formats? I don't think so, because for PKI, "auth_version" seems not used(self.verify_signed_token called).
Agree to Andrea.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-cinderclient (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/80563

melanie witt (melwitt)
Changed in python-novaclient:
assignee: nobody → Melanie Witt (melwitt)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Jordan Pittier (jordan-pittier) wrote :

I am also hitting this. Two workarounds worked for me (Icehouse, Keystone v2) :
Either set auth_version = v2.0
!Or! set auth_port = 5000

Revision history for this message
melanie witt (melwitt) wrote :

I think this patch would allow novaclient to work with keystone v3:

https://review.openstack.org/#/c/85920/

Changed in python-novaclient:
assignee: Melanie Witt (melwitt) → nobody
Revision history for this message
Edward Chapin (edward-chapin) wrote :

I'm trying to get the dashboard working with multi-domains with IceHouse. Basically it looks like the same sort of thing - see this post https://ask.openstack.org/en/question/45872/icehouse-dashboard-problems-using-multi-domain-support/. It seems to authenticate with a domain-scoped token, but when it tries to query nova for resources I get a 401. I've tried explicitly setting auth_version = v2.0 in nova.conf to no avail... Is this related, or something else?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on python-cinderclient (master)

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/80563

Revision history for this message
melanie witt (melwitt) wrote :

I think this has been fixed by:

https://review.openstack.org/#/c/85920/
https://review.openstack.org/#/c/105900/

I tested in devstack by:

OS_PROJECT_DOMAIN_ID=default
OS_REGION_NAME=RegionOne
OS_USER_DOMAIN_ID=default
OS_IDENTITY_API_VERSION=3.0
OS_AUTH_URL=http://10.0.2.15:5000/v3
OS_USERNAME=demo
OS_TENANT_NAME=demo

$ nova boot --image cirros-0.3.2-x86_64-uec --flavor m1.tiny --poll test
$ cinder create 4
$ nova volume-attach 6af88e0d-e208-4364-8881-c8b2e4166292 7fbaa508-adb7-4357-a9e4-7cb656b2e5c1 /dev/vdb

Changed in python-novaclient:
status: Confirmed → Fix Committed
Michael Still (mikal)
Changed in python-novaclient:
milestone: none → 2.21.0
Michael Still (mikal)
Changed in python-novaclient:
status: Fix Committed → Fix Released
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

I don't believe this is an issue with Cinder anymore. If it is, please reopen.

Changed in python-cinderclient:
status: New → Invalid
Revision history for this message
Steve Martinelli (stevemar) wrote :

finding the service catalog through python bindings is available now, following sean's comment i don't believe this is an issue any longer. if it is, please re-open

Changed in python-keystoneclient:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.