OpenStack Compute (nova)

cinderclient AmbiguousEndpoints in Nova API when deleting nested stack

Bug #1289135 reported by Mike Spreitzer on 2014-03-07

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Invalid	Undecided	Unassigned
	python-cinderclient	Invalid	Undecided	Unassigned

Bug Description

While chasing down some errors I found the first one was the following, found in the log from the Nova API process.

2014-03-06 22:17:41.713 ERROR nova.api.openstack [req-0a2e7b6b-8ea8-48f1-b6c9-4c6a20ba27b4 admin admin] Caught error: AmbiguousEndpoints: [{u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'admin', u'id': u'154c830dce20478a8b269b5f85f7bca3'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'public', u'id': u'4129f440fa42491f997984455b9727af'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'internal', u'id': u'7f2013973d0248f1ba64ece67e3df7bb'}]
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack Traceback (most recent call last):
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return req.get_response(self.application)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack application, catch_exc_info=False)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 596, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return self.app(env, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack response = self.app(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return resp(environ, start_response)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 925, in __call__
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack content_type, body, accept)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 987, in _process_stack
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1074, in dispatch
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack return method(req=request, **action_args)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/contrib/volumes.py", line 502, in delete
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack volume = self.volume_api.get(context, volume_id)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/volume/cinder.py", line 174, in wrapper
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack res = method(self, ctx, volume_id, *args, **kwargs)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/volume/cinder.py", line 207, in get
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack item = cinderclient(context).volumes.get(volume_id)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/nova/nova/volume/cinder.py", line 92, in cinderclient
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack endpoint_type=endpoint_type)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack File "/opt/stack/python-cinderclient/cinderclient/service_catalog.py", line 83, in url_for
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack endpoints=matching_endpoints)
2014-03-06 22:17:41.713 17339 TRACE nova.api.openstack AmbiguousEndpoints: AmbiguousEndpoints: [{u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'admin', u'id': u'154c830dce20478a8b269b5f85f7bca3'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'public', u'id': u'4129f440fa42491f997984455b9727af'}, {u'url': u'http://10.10.0.24:8776/v1/a4c140dd5649439987f9c61d0a91c76e', u'region': u'RegionOne', u'legacy_endpoint_id': u'58ac5510148c4641ab48e1499c0bb4ec', 'serviceName': None, u'interface': u'internal', u'id': u'7f2013973d0248f1ba64ece67e3df7bb'}]

This is reminiscent of bug 1154809 --- which was fixed a year ago.

My scenario is as follows. I started with a fresh DevStack install around 04:00 on March 5, 2013 UTC. Heat Orchestration Template VM_with_EBS.yaml has three resources: a VM instance, a volume, and an attachment between the two. HOT ASRG2.yaml has several resources, one of which is an OS::Heat::AutoScalingGroup whose element type is a stack made from VM_with_EBS.yaml. FYI, an OS::Heat::AutoScalingGroup is implemented by a nested stack. Thus, there are two levels of nesting: VM_with_EBS within the Heat ASG, and the Heat ASG within ASRG2. Creating all this works fine. When I POST to the URL for scaling down the OS::Heat::AutoScalingGroup, that is when the AmbiguousEndpoints exception happens.

I am attaching VM_with_EBS.yaml with the initial bug report; I will attach ASRG2.yaml when I figure out how to do so.

Tags:

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-07:

The Heat Orchestration Template for the innermost stacks Edit (1.4 KiB, text/plain)

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-07:

HOT for the outermost stack Edit (2.9 KiB, text/plain)

Tracy Jones (tjones-i) on 2014-03-12

tags:

added: api

Revision history for this message

John Garbutt (johngarbutt) wrote on 2014-03-12:

Seems like devstack doesn't configure keystone in a way that heat recognises

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-13:

Download full text (3.5 KiB)

I did a fresh DevStack install tonight and the problem remains.

John, it is the Nova API process that is confounded by the Cinder endpoints; many Heat operations work just fine. Where would I look to track down a mismatch between Nova and Keystone or Cinder?

Here are some potentially relevant things I found in /etc/nova/nova.conf:

[DEFAULT]
...
auth_strategy = keystone
...

[osapi_v3]
enabled = True

Here are excerpts from the output from `keystone catalog`:

Service: identity
+-------------+----------------------------------+
| Property | Value |...

I did a fresh DevStack install tonight and the problem remains.

John, it is the Nova API process that is confounded by the Cinder endpoints; many Heat operations work just fine.  Where would I look to track down a mismatch between Nova and Keystone or Cinder?

Here are some potentially relevant things I found in /etc/nova/nova.conf:

[DEFAULT]
...
auth_strategy = keystone
...

[osapi_v3]
enabled = True

Here are excerpts from the output from `keystone catalog`:

Revision history for this message

Christopher Yeoh (cyeoh-0) wrote on 2014-03-19:

Could you post your settings for the following nova.conf settings:

os_region_name
cinder_endpoint_template
cinder_catalog_info

Thanks!

Chris

Changed in nova:
status:	New → Incomplete

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-19:

archive of various conf files Edit (29.0 KiB, application/zip)

Chris, none of those things is in /etc/nova/nova.conf !

I have attached an archive of the nova, keystone, and cinder conf files from /etc

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-19:

I answered the request for more information.

Changed in nova:
status:	Incomplete → New

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-20:

This reminds me of bug 1263876

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-21:

This bug is still present in

commit 96f427b592ee0264ad9553592878718e86c799f7
Merge: f17362a 7c833ec
Author: Jenkins <email address hidden>
Date: Tue Mar 18 22:58:34 2014 +0000

Merge "Allow update of disable_rollback"

Revision history for this message

Christopher Yeoh (cyeoh-0) wrote on 2014-03-25:

#10

Thanks Mike - any chance you could attach the output from keystone service-list and endpoint-list ?

Adding python-cinderclient as I'm guessing the problem might be in there somewhere.

Revision history for this message

Mike Spreitzer (mike-spreitzer) wrote on 2014-03-25:

#11

keystone service-list and endpoint-list Edit (4.8 KiB, text/plain)

Attached requested info

Revision history for this message

Jordan Pittier (jordan-pittier) wrote on 2014-05-05:

#12

I am also hitting this.

cinder_api_insecure = True
cinder_catalog_info = volume:cinder:internalURL

New info : this works :
auth_uri = https://identity.cw-labs.net/v2.0
auth_host = identity.usr.lab0.aub.cw-labs.net
auth_port = 5000

This doesn't work (raise AmbiguousEndpoints)
auth_uri = https://identity.cw-labs.net/v2.0
auth_host = identity.usr.lab0.aub.cw-labs.net
auth_port = 35357

Revision history for this message

Sean Dague (sdague) wrote on 2014-09-10:

#13

Assuming this is in the cinder client

Changed in nova:
status:	New → Invalid

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2015-11-03: Bug Cleanup

#14

Closing stale bug. If this is still an issue please reopen.

Changed in python-cinderclient:
status:	New → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.