problems terminating lxc instances

[James Page]

I was not able to consistently terminate lxc instances using the following command:

euca-terminate-instance i-XXXXX

I was able to quite rapidly get to a point where I was unable to start any new instances because I had to many in the 'running' state - non of which where actually running.

Error from nova-compute log file:

2011-09-06 15:17:01,765 DEBUG nova.utils [-] Running cmd (subprocess): sudo umount /var/lib/nova/instances/instance-00000002/rootfs from (pid=1229) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-06 15:17:01,774 INFO nova.virt.libvirt_conn [-] Instance instance-00000002 destroyed successfully.
2011-09-06 15:17:01,929 DEBUG nova.utils [-] Result was 1 from (pid=1229) execute /usr/lib/pymodules/python2.7/nova/utils.py:180
2011-09-06 15:17:01,930 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup -a from (pid=1229) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-06 15:17:02,033 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 117, in decorated_function
(nova.exception): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 492, in terminate_instance
(nova.exception): TRACE: self._shutdown_instance(context, instance_id, 'Terminating')
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 483, in _shutdown_instance
(nova.exception): TRACE: self.driver.destroy(instance, network_info)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 326, in destroy
(nova.exception): TRACE: self._cleanup(instance)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 336, in _cleanup
(nova.exception): TRACE: disk.destroy_container(target, instance, nbd=FLAGS.use_cow_images)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/disk.py", line 153, in destroy_container
(nova.exception): TRACE: utils.execute('umount', container_dir, run_as_root=True)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/utils.py", line 188, in execute
(nova.exception): TRACE: cmd=' '.join(cmd))
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: sudo umount /var/lib/nova/instances/instance-00000002/rootfs
(nova.exception): TRACE: Exit code: 1
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: 'umount: /var/lib/nova/instances/instance-00000002/rootfs: not mounted\n'
(nova.exception): TRACE:
2011-09-06 15:17:02,049 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: Unexpected error while running command.
(nova.rpc): TRACE: Command: sudo umount /var/lib/nova/instances/instance-00000002/rootfs
(nova.rpc): TRACE: Exit code: 1
(nova.rpc): TRACE: Stdout: ''

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: nova-compute-lxc 2011.3~rc~20110901.1523-0ubuntu1
ProcVersionSignature: Ubuntu 3.0.0-10.16-server 3.0.4
Uname: Linux 3.0.0-10-server x86_64
Architecture: amd64
Date: Tue Sep 6 16:35:53 2011
NovaConf: Error: [Errno 13] Permission denied: '/etc/nova/nova.conf'
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_GB:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: nova
UpgradeStatus: No upgrade log present (probably fresh install)

[James Page (james-page)]

Picked this out of a clean install:

2011-09-06 17:29:26,520 DEBUG nova.virt.libvirt.firewall [-] The nwfilter(nova-instance-instance-00000001-secgroup) for instance-00000001 is not found. from (pid=1164) unfilter_instance /usr/lib/pymodules/python2.7/nova/virt/libvirt/firewall.py:320
2011-09-06 17:29:26,521 INFO nova.virt.libvirt_conn [-] instance instance-00000001: deleting instance files /var/lib/nova/instances/instance-00000001
2011-09-06 17:29:26,521 DEBUG nova.utils [-] Running cmd (subprocess): sudo umount /var/lib/nova/instances/instance-00000001/rootfs from (pid=1164) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-06 17:29:26,544 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup -a from (pid=1164) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-06 17:29:26,622 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 117, in decorated_function
(nova.exception): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 492, in terminate_instance
(nova.exception): TRACE: self._shutdown_instance(context, instance_id, 'Terminating')
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 483, in _shutdown_instance
(nova.exception): TRACE: self.driver.destroy(instance, network_info)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 326, in destroy
(nova.exception): TRACE: self._cleanup(instance)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 338, in _cleanup
(nova.exception): TRACE: shutil.rmtree(target)
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 244, in rmtree
(nova.exception): TRACE: rmtree(fullname, ignore_errors, onerror)
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 253, in rmtree
(nova.exception): TRACE: onerror(os.rmdir, path, sys.exc_info())
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 251, in rmtree
(nova.exception): TRACE: os.rmdir(path)
(nova.exception): TRACE: OSError: [Errno 16] Device or resource busy: '/var/lib/nova/instances/instance-00000001/rootfs'
(nova.exception): TRACE:
2011-09-06 17:29:26,661 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: [Errno 16] Device or resource busy: '/var/lib/nova/instances/instance-00000001/rootfs'
(nova.rpc): TRACE:

[James Page (james-page)]

Instance remained 'running' - further attempts to terminate it resulted in the error in the original bug report

I'm using these test instructions with modifications for LXC to setup a single node install:

 https://wiki.ubuntu.com/ServerTeam/Oneiric/OpenStackTestPlan

[Clint Byrum (clint-fewbar)]

Marking High importance in Ubuntu, as this basically renders LXC unusable on nova. Probably an upstream bug, so adding an also-affects.

[Chuck Short (zulcss)]

This should be in the new snapshot for Ubuntu.

[Thierry Carrez (ttx)]

No fix committed in nova ?

[Dave Walker (davewalker)]

Marking fix released for Ubuntu task, as zul believes this to be resolved. Please re-open if this is not the case, thanks.