LXC instance failed to terminate

[Samuel Hassine]

Hi,

I am using the Nova version:

ii nova-compute 2012.1~e4-1 OpenStack Compute - compute node
ii nova-compute-lxc 2012.1~e4-1 OpenStack Compute - compute node (LXC)

Sometimes, the LXC instance does not terminate, here is the trace in the log:

(nova.rpc.common): TRACE: Traceback (most recent call last):
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 250, in _process_data
(nova.rpc.common): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 112, in wrapped
(nova.rpc.common): TRACE: return f(*args, **kw)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 156, in decorated_function
(nova.rpc.common): TRACE: function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 180, in decorated_function
(nova.rpc.common): TRACE: sys.exc_info())
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 174, in decorated_function
(nova.rpc.common): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 736, in terminate_instance
(nova.rpc.common): TRACE: self._delete_instance(context, instance)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 716, in _delete_instance
(nova.rpc.common): TRACE: self._shutdown_instance(context, instance, 'Terminating')
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 687, in _shutdown_instance
(nova.rpc.common): TRACE: block_device_info)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 425, in destroy
(nova.rpc.common): TRACE: cleanup=True)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 419, in _destroy
(nova.rpc.common): TRACE: self._cleanup(instance)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 435, in _cleanup
(nova.rpc.common): TRACE: shutil.rmtree(target)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/shutil.py", line 245, in rmtree
(nova.rpc.common): TRACE: rmtree(fullname, ignore_errors, onerror)
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
(nova.rpc.common): TRACE: onerror(os.rmdir, path, sys.exc_info())
(nova.rpc.common): TRACE: File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
(nova.rpc.common): TRACE: os.rmdir(path)
(nova.rpc.common): TRACE: OSError: [Errno 16] Device or resource busy: '/var/lib/nova/instances/instance-0000000f/rootfs'
(nova.rpc.common): TRACE:

After, it is impossible to manually remove /var/lib/nova/instances/instance-0000000f/rootfs

Nothing in lsof | grep 0000000f, the only way is to physically reboot the server...

Best regards.
Samuel