VMware: Detaching volume fails with FileNotFoundException

Bug #1425502 reported by Vipin Balachandran
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Gary Kotton

Bug Description

Steps to reproduce:

a) Create nova instance vm-1 (assume that the datastore is a shared datastore 'ds-1' which is a member of datastore cluster dcls-1)
b) Create volume vol-1 with a storage profile which maps to ds-1
c) Attach vol-1 to vm-1
d) Migrate vm-1's datastore to shared datastore ds-2 which is a member of the dcls-1
(The vmdk of vol-1 *moves* to vm-1's new location after this step)
e) Detach vol-1

2015-02-25 18:05:00.014 ERROR oslo_messaging.rpc.dispatcher [req-124c1c0a-2a00-46b1-a790-0a2e5d6da871 admin demo] Exception during message handling: File /vmfs/volumes/54ae58ae-5db2e67f-38b1-0200066a0432/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19.vmdk was not found
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 416, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher payload)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 300, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher pass
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 285, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 328, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 316, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4721, in detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher self._detach_volume(context, instance, bdm)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4690, in _detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher self.volume_api.roll_detaching(context, volume_id)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4678, in _detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher encryption=encryption)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 487, in detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher instance)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 539, in detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher self._detach_volume_vmdk(connection_info, instance)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 504, in _detach_volume_vmdk
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher disk_type=vmdk.disk_type)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 455, in _consolidate_vmdk_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher self._relocate_vmdk_volume(volume_ref, res_pool, datastore, host)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 384, in _relocate_vmdk_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher self._session._wait_for_task(task)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 673, in _wait_for_task
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return self.wait_for_task(task_ref)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 380, in wait_for_task
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return evt.wait()
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return hubs.get_hub().switch()
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher return self.greenlet.switch()
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 76, in _inner
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher self.f(*self.args, **self.kw)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 417, in _poll_task
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher raise task_ex
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher FileNotFoundException: File /vmfs/volumes/54ae58ae-5db2e67f-38b1-0200066a0432/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19.vmdk was not found
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher

Changed in nova:
assignee: nobody → Vipin Balachandran (vbala)
Changed in nova:
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/159738

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
assignee: Vipin Balachandran (vbala) → Gary Kotton (garyk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/159738
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=24488387a1e1817e82a1a7a3c1d9d9c7bdf5d165
Submitter: Jenkins
Branch: master

commit 24488387a1e1817e82a1a7a3c1d9d9c7bdf5d165
Author: Vipin Balachandran <email address hidden>
Date: Thu Feb 5 16:37:38 2015 +0530

    VMware: Handle missing vmdk during volume detach

    During volume detach, volume's shadow VM might need to
    be relocated if storage DRS moved the Nova instance along
    with volume's vmdk. The shadow VM relocate will fail with
    FileNotFoundException if SDRS moved the volume vmdk instead
    of copying it to the new location. This patch fixes the
    problem by removing the virtual disk device of shadow VM if
    relocate failed due to missing vmdk file. After removing the
    device, relocate is attempted again.

    Closes-Bug: #1425502
    Change-Id: I40a25abe1ce96f1dc7d78ec46c948c8db451632b

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-3 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.