Detach VMDK volume fails if shadow VM unregistered and registered back

[Chinmaya Bharadwaj]

Description
===========
When trying nova VMDK volume detach from an instance hosted on vCenter, the detach fail if the shadow VM used for vmdk volume is unregistered and registered back in vCenter.

Steps to reproduce
==================

* Attach a VMDK volume to an instance
* Unregister/delete the shadow VM. [A VM just contains the VMDK matching the cinder UUID]
* register/create the VM back with the same name
* Try detach

Expected result
===============
It should detach the volume from the instance

Actual result
=============

It fails to detach.

Because in the vCenter, due to un-registering the vm MOid changed. In block_device_mapping table we store
this information (connection_info) and use it to form the volume ref during detach in the driver.

https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L305

I think we should not be relying on that, instead make the vCenter call and get the MOID by name.

Logs
====
Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
Faults: [ManagedObjectNotFound]
Details: {'obj': 'vm-1031'}
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher payload)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 387, in decorated_function
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4892, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher attachment_id=attachment_id)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4835, in _detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher connection_info = self._driver_detach_volume(context, instance, bdm)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4787, in _driver_detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.volume_api.roll_detaching(context, volume_id)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4775, in _driver_detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher encryption=encryption)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 391, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._volumeops.detach_volume(connection_info, instance)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 582, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self._detach_volume_vmdk(connection_info, instance)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 530, in _detach_volume_vmdk
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher vmdk = "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 635, in get_vmdk_info
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher "config.hardware.device")
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 596, in _call_method
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.invoke_api(module, method, self.vim, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 348, in invoke_api
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return _invoke_api(module, method, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return evt.wait()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return hubs.get_hub().switch()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.greenlet.switch()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher idle = self.f(*self.args, **self.kw)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = f(*args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 332, in _invoke_api
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher details=excep.details)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Faults: [ManagedObjectNotFound]
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Details: {'obj': 'vm-1031'}
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher

Environment
===========

nova driver: VMWareVCDriver
cinder driver: VMDkdriver
Observing the issue on "stable/mitaka". I believe this issue persists in master as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/332156

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/332156
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[Sean Dague (sdague)]

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'