This could be related, libvirt domain delete/destroy fails because the backing volume is busy:
http://logs.openstack.org/20/128020/2/check/check-tempest-dsvm-full/6b56bf1/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-10-13_18_12_06_653
2014-10-13 18:12:06.653 ERROR nova.virt.libvirt.driver [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 16365 with SIGKILL: Device or resource busy 2014-10-13 18:12:08.000 ERROR nova.compute.manager [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Setting instance vm_state to ERROR 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] Traceback (most recent call last): 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2500, in do_terminate_instance 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] self._delete_instance(context, instance, bdms, quotas) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/hooks.py", line 131, in inner 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] rv = f(*args, **kwargs) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2469, in _delete_instance 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] quotas.rollback() 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__ 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(self.type_, self.value, self.tb) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2446, in _delete_instance 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] self._shutdown_instance(context, instance, bdms) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2376, in _shutdown_instance 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] requested_networks) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__ 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(self.type_, self.value, self.tb) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/compute/manager.py", line 2365, in _shutdown_instance 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] block_device_info) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1054, in destroy 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] self._destroy(instance) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1010, in _destroy 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] instance=instance) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__ 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(self.type_, self.value, self.tb) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 979, in _destroy 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] virt_dom.destroy() 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] result = proxy_call(self._autowrap, f, *args, **kwargs) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] rv = execute(f, *args, **kwargs) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] six.reraise(c, e, tb) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] rv = meth(*args, **kwargs) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 918, in destroy 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self) 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] libvirtError: Failed to terminate process 16365 with SIGKILL: Device or resource busy 2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]
This could be related, libvirt domain delete/destroy fails because the backing volume is busy:
http:// logs.openstack. org/20/ 128020/ 2/check/ check-tempest- dsvm-full/ 6b56bf1/ logs/screen- n-cpu.txt. gz?level= TRACE#_ 2014-10- 13_18_12_ 06_653
2014-10-13 18:12:06.653 ERROR nova.virt. libvirt. driver [req-017c0b4b- 11ba-4f59- 9895-3d0053481b 00 VolumesV2Action sTestXML- 1527860598 VolumesV2Action sTestXML- 196352251] [instance: 6f733615- 3124-400e- b890-b07468fa47 77] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 16365 with SIGKILL: Device or resource busy manager [req-017c0b4b- 11ba-4f59- 9895-3d0053481b 00 VolumesV2Action sTestXML- 1527860598 VolumesV2Action sTestXML- 196352251] [instance: 6f733615- 3124-400e- b890-b07468fa47 77] Setting instance vm_state to ERROR manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] Traceback (most recent call last): manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2500, in do_terminate_ instance manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] self._delete_ instance( context, instance, bdms, quotas) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/hooks. py", line 131, in inner manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] rv = f(*args, **kwargs) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2469, in _delete_instance manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] quotas.rollback() manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ oslo/utils/ excutils. py", line 82, in __exit__ manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] six.reraise( self.type_ , self.value, self.tb) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2446, in _delete_instance manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] self._shutdown_ instance( context, instance, bdms) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2376, in _shutdown_instance manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] requested_networks) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ oslo/utils/ excutils. py", line 82, in __exit__ manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] six.reraise( self.type_ , self.value, self.tb) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2365, in _shutdown_instance manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] block_device_info) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 1054, in destroy manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] self._destroy( instance) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 1010, in _destroy manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] instance=instance) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ oslo/utils/ excutils. py", line 82, in __exit__ manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] six.reraise( self.type_ , self.value, self.tb) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 979, in _destroy manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] virt_dom.destroy() manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 183, in doit manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] result = proxy_call( self._autowrap, f, *args, **kwargs) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 141, in proxy_call manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] rv = execute(f, *args, **kwargs) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 122, in execute manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] six.reraise(c, e, tb) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 80, in tworker manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] rv = meth(*args, **kwargs) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] File "/usr/lib/ python2. 7/dist- packages/ libvirt. py", line 918, in destroy manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] if ret == -1: raise libvirtError ('virDomainDest roy() failed', dom=self) manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77] libvirtError: Failed to terminate process 16365 with SIGKILL: Device or resource busy manager [instance: 6f733615- 3124-400e- b890-b07468fa47 77]
2014-10-13 18:12:08.000 ERROR nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.
2014-10-13 18:12:08.000 17350 TRACE nova.compute.