OpenStack Compute (nova)

Unable to delete instance when volume detached at the same time

Bug #1815792 reported by Gary Kotton on 2019-02-13

This bug report is a duplicate of: Bug #1662483: detach_volume races with delete. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	In Progress	Undecided	Gary Kotton

Bug Description

When using K8s above OpenStack there is a race condition when a persistant volume will be deleted at the same time that a instance using that volume is deleted. This results in the instance going into an error state.

201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] Traceback (most recent call last):
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2478, in do_terminate_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self._delete_instance(context, instance, bdms, quotas)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] rv = f(*args, **kwargs)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2441, in _delete_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] quotas.rollback()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self.force_reraise()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] six.reraise(self.type_, self.value, self.tb)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2405, in _delete_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self._shutdown_instance(context, instance, bdms)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self._detach_volume_vmdk(connection_info, instance)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 609, in _detach_volume_vmdk
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self.detach_disk_from_vm(vm_ref, instance, device)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 157, in detach_disk_from_vm
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] vm_util.reconfigure_vm(self._session, vm_ref, vmdk_detach_config_spec)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vm_util.py", line 2681, in reconfigure_vm
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] session._wait_for_task(reconfig_task)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 719, in _wait_for_task
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] return self.wait_for_task(task_ref)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 398, in wait_for_task
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] return evt.wait()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] return hubs.get_hub().switch()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] return self.greenlet.switch()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self.f(*self.args, **self.kw)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 454, in _poll_task
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] raise task_ex
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] VimFaultException: Invalid configuration for device '0'.
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] Faults: ['InvalidDeviceSpec']
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]

201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2305, in _shutdown_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] requested_networks)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self.force_reraise()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] six.reraise(self.type_, self.value, self.tb)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2292, in _shutdown_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] block_device_info)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 564, in destroy
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self._detach_instance_volumes(instance, block_device_info)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 539, in _detach_instance_volumes
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] instance=instance)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self.force_reraise()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] six.reraise(self.type_, self.value, self.tb)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 528, in _detach_instance_volumes
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] disk.get('device_name'))
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 470, in detach_volume
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] self._volumeops.detach_volume(connection_info, instance)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 663, in detach_volume

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-13: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/636701

Changed in nova:
assignee:	nobody → Gary Kotton (garyk)
status:	New → In Progress

Revision history for this message

Matt Riedemann (mriedem) wrote on 2019-02-13:

This was already fixed at one point with https://review.openstack.org/#/c/430213/ but was abandoned due to inactivity (it needed to be rebased). I've restored it so you can just rebase it.

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1662483 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.