OpenStack Compute (nova)

VMware: rebuild an instance failed

Bug #1563684 reported by Dongcan Ye
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Medium
Unassigned

Bug Description

Description
===================
Nova version: master
Virt driver: VCDriver

Steps to reproduce
=====================
1. Nova boot an instance.
2. Cinder create a volume.
3. Attach volume to the instance.
4. After attached success, using nova image-create command, create a snapshot for the instance.
5. Rebuild the instance with snapshot image.

In nova-compute, error info:
2016-03-30 12:16:14.801 9290 DEBUG oslo_vmware.exceptions [-] Fault GenericVmConfigFault not matched. get_fault_class /usr/lib/python2.7/site-packages/oslo_vmware/exceptions.py:250
2016-03-30 12:16:14.801 9290 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall Traceback (most recent call last):
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall File "/usr/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall self.f(*self.args, **self.kw)
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall File "/usr/lib/python2.7/site-packages/oslo_vmware/api.py", line 417, in _poll_task
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall raise task_ex
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall VMwareDriverException: Disk '/vmfs/volumes/546a3532-ca1f27f1-e66a-1458d04cf670/volume-dbbdd17d-bb5c-4804-9d30-a6bd8a96eca3/volume-dbbdd17d-bb5c-4804-9d30-a6bd8a96eca3.vmdk' cannot be opened for writing. It might be shared with some other VM.
2016-03-30 12:16:14.801 9290 TRACE oslo_vmware.common.loopingcall
2016-03-30 12:16:14.802 9290 ERROR nova.compute.manager [req-b0c16cc2-fd52-4c85-9ff9-c67ff68ee410 4412e38ec9814b96a03e63097ec51f1a 8f75187cd29f4715881f450646fc6e08 - - -] [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] Setting instance vm_state to ERROR
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] Traceback (most recent call last):
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6418, in _error_out_instance_on_exception
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] yield
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3087, in rebuild_instance
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] self._rebuild_default_impl(**kwargs)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2934, in _rebuild_default_impl
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] block_device_info=new_block_device_info)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 481, in spawn
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] admin_password, network_info, block_device_info)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 683, in spawn
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] vm_util.power_on_instance(self._session, instance, vm_ref=vm_ref)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vm_util.py", line 1394, in power_on_instance
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] session._wait_for_task(poweron_task)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 681, in _wait_for_task
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] return self.wait_for_task(task_ref)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/oslo_vmware/api.py", line 380, in wait_for_task
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] return evt.wait()
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] return hubs.get_hub().switch()
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] return self.greenlet.switch()
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] self.f(*self.args, **self.kw)
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] File "/usr/lib/python2.7/site-packages/oslo_vmware/api.py", line 417, in _poll_task
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] raise task_ex
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05] VMwareDriverException: Disk '/vmfs/volumes/546a3532-ca1f27f1-e66a-1458d04cf670/volume-dbbdd17d-bb5c-4804-9d30-a6bd8a96eca3/volume-dbbdd17d-bb5c-4804-9d30-a6bd8a96eca3.vmdk' cannot be opened for writing. It might be shared with some other VM.
2016-03-30 12:16:14.802 9290 TRACE nova.compute.manager [instance: e338fd04-859f-4fa4-8f1d-cd2e297b0c05]

Tags: rebuild vmware
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Could you please let us know if that issue is also reproducible on the previous stable version, ie. Liberty ?

Changed in nova:
importance: Undecided → High
importance: High → Medium
Revision history for this message
Dongcan Ye (hellochosen) wrote :

@Sylvain, I also test in Kilo, it can reproduced. For Liberty, I had not tested, maybe the issue can reproduced.

Revision history for this message
Tracy Jones (tjones-i) wrote :

vipin can you take a look?

Changed in nova:
assignee: nobody → Giridhar Jayavelu (gjayavelu)
assignee: Giridhar Jayavelu (gjayavelu) → nobody
assignee: nobody → Vipin Balachandran (vbala)
Revision history for this message
Gary Kotton (garyk) wrote :

@Slvain - this patch most probably address this - https://review.openstack.org/#/c/218639/. So can you please review.

Matt Riedemann (mriedem)
Changed in nova:
status: New → Confirmed
Sarafraj Singh (sarafraj-singh)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
assignee: Vipin Balachandran (vbala) → nobody
status: In Progress → Confirmed
Sean Dague (sdague)
Changed in nova:
assignee: nobody → Vipin Balachandran (vbala)
status: Confirmed → In Progress
Matt Riedemann (mriedem)
Changed in nova:
status: In Progress → Confirmed
assignee: Vipin Balachandran (vbala) → nobody
tags: added: rebuild
Sushma Gunda (sushma05)
Changed in nova:
assignee: nobody → Sushma Gunda (sushma05)
status: Confirmed → In Progress
Sushma Gunda (sushma05)
Changed in nova:
assignee: Sushma Gunda (sushma05) → nobody
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.