Comment 5 for bug 1373513

Revision history for this message
Matt Riedemann (mriedem) wrote :

This could be related, libvirt domain delete/destroy fails because the backing volume is busy:

http://logs.openstack.org/20/128020/2/check/check-tempest-dsvm-full/6b56bf1/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-10-13_18_12_06_653

2014-10-13 18:12:06.653 ERROR nova.virt.libvirt.driver [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 16365 with SIGKILL: Device or resource busy
2014-10-13 18:12:08.000 ERROR nova.compute.manager [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Setting instance vm_state to ERROR
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] Traceback (most recent call last):
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2500, in do_terminate_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] self._delete_instance(context, instance, bdms, quotas)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/hooks.py", line 131, in inner
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] rv = f(*args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2469, in _delete_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] quotas.rollback()
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(self.type_, self.value, self.tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2446, in _delete_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] self._shutdown_instance(context, instance, bdms)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2376, in _shutdown_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] requested_networks)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(self.type_, self.value, self.tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2365, in _shutdown_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] block_device_info)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1054, in destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] self._destroy(instance)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1010, in _destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] instance=instance)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(self.type_, self.value, self.tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 979, in _destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] virt_dom.destroy()
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] rv = execute(f, *args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(c, e, tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] rv = meth(*args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 918, in destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] libvirtError: Failed to terminate process 16365 with SIGKILL: Device or resource busy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]