vmwareapi: unrescue instance failed due to can't detach disk from running instance

[dingxy]

I use openstack icehouse build, when unrescue a rescued instance based on vcenter driver, the instance became to ERROR status, and when nova show we hit error below:
{u'message': u'The attempted operation cannot be performed in the current state (Powered on).', u'code': 500, u'details': u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2797, in unrescue_instance |
| | network_info) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 675, in unrescue |
| | _vmops.unrescue(instance) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1132, in unrescue |
| | self._volumeops.detach_disk_from_vm(vm_rescue_ref, r_instance, device) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/volumeops.py", line 129, in detach_disk_from_vm |
| | self._session._wait_for_task(instance_uuid, reconfig_task) |
| | File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 890, in _wait_for_task |
| | ret_val = done.wait() |
| | File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait |
| | return hubs.get_hub().switch() |
| | File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch |
| | return self.greenlet.switch()

[ChangBo Guo(gcb) (glongwave)]

The roort cause is that we can't detach Non-Hot Plug disk from running instance, a proposed fix like:
https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vmops.py#L1132
   1131 device = vm_util.get_vmdk_volume_disk(hardware_devices, path=vmdk_path)
   1132 +self.power_off(instance, vm_ref=vm_rescue_ref)
   1133 self._volumeops.detach_disk_from_vm(vm_rescue_ref, r_instance, device)
   1134 self.destroy(r_instance, None, instance_name=instance_name)
   1135 self._power_on(instance)
   1136
   1137 +def power_off(self, instance, vm_ref=None):
   1138 """Power off the specified instance."""
   1139 + if not vm_ref:
   1140 + vm_ref = vm_util.get_vm_ref(self._session, instance)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/75788

[Gary Kotton (garyk)]

Hi,
Can you please clarify what a "Non-Hot Plug" disk is? In addition to this can you please provide the steps to reproduce the problem.
I am unable to reproduce. What I do is the following:
1. deploy a vm - nova boot --image debian-2.6.32-i686 --flavor 1 X
2. rescue the vm - nova rescue X
3. unrescue the vm - nova unrescue X
I did not see the problem reported. Can you please explain what is missing?
Thanks
Gary

[dingxy (xyding)]

the step is same with yours, the difference I think is the image, I use image trend-tinyvm1-flat.vmdk, and import it with command:"glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk".
 "Non-Hot Plug" disk mean the disk could not be detached from the VM when the VM is Active, the disk can only be detached when the vm is shutdown. I know "IDE" is one one of the disk type that is "Non-Hot Plug" disk.

[Mark McLoughlin (markmc)]

Adding to icehouse-rc-potential because bug #1269418 is already tagged with icehouse-rc-potential and apparently it requires this bug to be fixed first

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/75788
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1e1915aaaca38b5691794e0e052a42b9d95dd3c2
Submitter: Jenkins
Branch: master

commit 1e1915aaaca38b5691794e0e052a42b9d95dd3c2
Author: Xiaoyan Ding <email address hidden>
Date: Mon Feb 24 16:17:46 2014 +0800

    VMWare: add power off vm before detach disk during unrescue

    Non Hot Plug type disk like IDE can only be detached when the VM is power off.

    Change-Id: Ib1f387a41abe2b52357854e90c2535ebb7b43f18
    Close-bug: #1279199

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/87088

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/88514

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/89762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/88514
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fb030283bed9e41a0343581fa21b81b2ebb07f15
Submitter: Jenkins
Branch: stable/icehouse

commit fb030283bed9e41a0343581fa21b81b2ebb07f15
Author: Xiaoyan Ding <email address hidden>
Date: Mon Feb 24 16:17:46 2014 +0800

    VMWare: add power off vm before detach disk during unrescue

    Non Hot Plug type disk like IDE can only be detached when the VM is power off.

    Change-Id: Ib1f387a41abe2b52357854e90c2535ebb7b43f18
    Close-bug: #1279199
    (cherry picked from commit 1e1915aaaca38b5691794e0e052a42b9d95dd3c2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/89762
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=36d892381178042ab8fe2f0b34a2fd03b2ac7014
Submitter: Jenkins
Branch: stable/havana

commit 36d892381178042ab8fe2f0b34a2fd03b2ac7014
Author: Xiaoyan Ding <email address hidden>
Date: Mon Feb 24 16:17:46 2014 +0800

    VMWare: add power off vm before detach disk during unrescue

    Non Hot Plug type disk like IDE can only be detached when the VM is power off.

    Close-bug: #1279199
    (cherry picked from commit 1e1915aaaca38b5691794e0e052a42b9d95dd3c2)

    Conflicts:

     nova/virt/vmwareapi/vmops.py

    Change-Id: Ib1f387a41abe2b52357854e90c2535ebb7b43f18