Detach VMDK volume fails if shadow VM unregistered and registered back

Bug #1593742 reported by Chinmaya Bharadwaj on 2016-06-17
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Unassigned

Bug Description

Description
===========
When trying nova VMDK volume detach from an instance hosted on vCenter, the detach fail if the shadow VM used for vmdk volume is unregistered and registered back in vCenter.

Steps to reproduce
==================

* Attach a VMDK volume to an instance
* Unregister/delete the shadow VM. [A VM just contains the VMDK matching the cinder UUID]
* register/create the VM back with the same name
* Try detach

Expected result
===============
It should detach the volume from the instance

Actual result
=============

It fails to detach.

Because in the vCenter, due to un-registering the vm MOid changed. In block_device_mapping table we store
this information (connection_info) and use it to form the volume ref during detach in the driver.

https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L305

I think we should not be relying on that, instead make the vCenter call and get the MOID by name.

Logs
====
Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
Faults: [ManagedObjectNotFound]
Details: {'obj': 'vm-1031'}
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher payload)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 387, in decorated_function
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4892, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher attachment_id=attachment_id)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4835, in _detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher connection_info = self._driver_detach_volume(context, instance, bdm)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4787, in _driver_detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.volume_api.roll_detaching(context, volume_id)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4775, in _driver_detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher encryption=encryption)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 391, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._volumeops.detach_volume(connection_info, instance)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 582, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self._detach_volume_vmdk(connection_info, instance)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 530, in _detach_volume_vmdk
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher vmdk = "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 635, in get_vmdk_info
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher "config.hardware.device")
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 596, in _call_method
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.invoke_api(module, method, self.vim, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 348, in invoke_api
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return _invoke_api(module, method, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return evt.wait()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return hubs.get_hub().switch()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.greenlet.switch()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher idle = self.f(*self.args, **self.kw)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = f(*args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 332, in _invoke_api
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher details=excep.details)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Faults: [ManagedObjectNotFound]
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Details: {'obj': 'vm-1031'}
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher

Environment
===========

nova driver: VMWareVCDriver
cinder driver: VMDkdriver
Observing the issue on "stable/mitaka". I believe this issue persists in master as well.

Changed in nova:
assignee: nobody → Chinmaya Bharadwaj (acbharadwaj)
Dongcan Ye (hellochosen) on 2016-06-18
tags: added: vmware

Fix proposed to branch: master
Review: https://review.openstack.org/332156

Changed in nova:
status: New → In Progress
Changed in nova:
assignee: Chinmaya Bharadwaj (acbharadwaj) → Divakar Padiyar Nandavar (divakar-padiyar-nandavar)
Changed in nova:
assignee: Divakar Padiyar Nandavar (divakar-padiyar-nandavar) → Chinmaya Bharadwaj (acbharadwaj)

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/332156
Reason: This review is > 6 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → New
assignee: Chinmaya Bharadwaj (acbharadwaj) → nobody
Sean Dague (sdague) wrote :

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

tags: added: openstack-version.mitaka
Sean Dague (sdague) on 2017-06-28
Changed in nova:
status: New → Confirmed
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers