EndpointNotFound when deleting volume backended instance

[Liyingjun]

When booting a volume backend instance, there may be an error because of volume creating error. and the following error occur when deleting the instance:

2014-07-22 11:19:15.305 14601 ERROR nova.compute.manager [-] [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] Failed to complete a deletion
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] Traceback (most recent call last):
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 845, in _init_instance
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] self._delete_instance(context, instance, bdms)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 103, in inner
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] rv = f(*args, **kwargs)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2220, in _delete_instance
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] user_id=user_id)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] six.reraise(self.type_, self.value, self.tb)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2190, in _delete_instance
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] self._shutdown_instance(context, db_inst, bdms)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2136, in _shutdown_instance
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] connector)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 174, in wrapper
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] res = method(self, ctx, volume_id, *args, **kwargs)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 276, in terminate_connection
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] return cinderclient(context).volumes.terminate_connection(volume_id,
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 92, in cinderclient
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] endpoint_type=endpoint_type)
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] File "/usr/lib/python2.7/dist-packages/cinderclient/service_catalog.py", line 80, in url_for
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] raise cinderclient.exceptions.EndpointNotFound()
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815] EndpointNotFound
2014-07-22 11:19:15.305 14601 TRACE nova.compute.manager [instance: dc0d5ad1-caa7-4160-9f6f-158ca1193815]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/108656

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/108656
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e5c3ff67b51f7c466d89778cc2c4999e10c7e08e
Submitter: Jenkins
Branch: master

commit e5c3ff67b51f7c466d89778cc2c4999e10c7e08e
Author: liyingjun <email address hidden>
Date: Thu Jun 26 21:48:52 2014 +0800

    EndpointNotFound deleting volume backend instance

    When there is some error in volume creating on booting from volume,
    we may get the EndpointNotFound exception when deleting the error instance.
    Then we can never delete the instance successfully. The exception should
    be ignored.

    Change-Id: Ia92b648eb18c70996747d361e227de771f33ee9c
    Closes-bug: 1346866

[Yaguang Tang (heut2008)]

I think the correct fix is to find why cinderclient raise EndpointNotFound , this is mostly caused by wrong configuration, can anyone reproduce this?

Li,
Can you give more details on how to reproduce this bug ? in what situation will you see this exception, thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/128166

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/128166
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/296890

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/296890
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.