Terminating an instance with --libvirt_type=lxc fails, can't detach loop device

Bug #841060 reported by Xavi Campa
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Chuck Short

Bug Description

Seems like the container is still running when nova tries to detach the container, therefore failing:

root@compute01:~# fuser -m /dev/loop1
/dev/loop1: 15525rce 15535rce 18189rce 18197rce 18203rce 18214rce

The processes:
15525 ? Ss 0:00 /sbin/init
15535 ? S 0:00 /sbin/plymouthd --mode=boot --attach-to-session
18189 ? S 0:00 upstart-udev-bridge --daemon
18197 ? Ss 0:00 /usr/sbin/sshd -D
18203 ? Sl 0:00 rsyslogd -c4
18214 ? S<s 0:00 udevd --daemon

Trace:
2011-09-04 16:18:31,704 INFO nova.virt.libvirt_conn [-] instance instance-00000110: deleting instance files /var/lib/nova/instances/instance-00000110
2011-09-04 16:18:31,706 DEBUG nova.utils [-] Running cmd (subprocess): sudo umount /var/lib/nova/instances/instance-00000110/rootfs from (pid=15094) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-04 16:18:31,867 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup -a from (pid=15094) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-04 16:18:32,033 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --detach /dev/loop1 from (pid=15094) execute /usr/lib/pymodules/python2.7/nova/utils.py:165
2011-09-04 16:18:32,184 DEBUG nova.utils [-] Result was 1 from (pid=15094) execute /usr/lib/pymodules/python2.7/nova/utils.py:180
2011-09-04 16:18:32,208 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 117, in decorated_function
(nova.exception): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 492, in terminate_instance
(nova.exception): TRACE: self._shutdown_instance(context, instance_id, 'Terminating')
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 483, in _shutdown_instance
(nova.exception): TRACE: self.driver.destroy(instance, network_info)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 326, in destroy
(nova.exception): TRACE: self._cleanup(instance)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 336, in _cleanup
(nova.exception): TRACE: disk.destroy_container(target, instance, nbd=FLAGS.use_cow_images)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/disk.py", line 159, in destroy_container
(nova.exception): TRACE: _unlink_device(device, nbd)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/disk.py", line 190, in _unlink_device
(nova.exception): TRACE: utils.execute('losetup', '--detach', device, run_as_root=True)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/utils.py", line 188, in execute
(nova.exception): TRACE: cmd=' '.join(cmd))
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: sudo losetup --detach /dev/loop1
(nova.exception): TRACE: Exit code: 1
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: "loop: can't delete device /dev/loop1: Device or resource busy\n"
(nova.exception): TRACE:
2011-09-04 16:18:32,218 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: Unexpected error while running command.
(nova.rpc): TRACE: Command: sudo losetup --detach /dev/loop1
(nova.rpc): TRACE: Exit code: 1
(nova.rpc): TRACE: Stdout: ''
(nova.rpc): TRACE: Stderr: "loop: can't delete device /dev/loop1: Device or resource busy\n"

Using these versions:
ii nova-common 2011.3~rc~20110903.1526-0ubuntu0ppa1~natty1 OpenStack Compute - common files
ii nova-compute 2011.3~rc~20110903.1526-0ubuntu0ppa1~natty1 OpenStack Compute - compute node
ii nova-compute-kvm 2011.3~rc~20110903.1526-0ubuntu0ppa1~natty1 OpenStack Compute - compute node (KVM)

My configuration:
--cc_host=controller
--ec2_url=http://controller:8773/services/Cloud
--ec2_host=controller
--rabbit_host=controller
--network_manager=nova.network.manager.VlanManager
--flat_interface=eth0
--fixed_range=172.16.0.0/16
--network_size=256
--sql_connection=mysql://nova:passW0rd@controller/nova
--glance_api_servers=glance:9292
--image_service=nova.image.glance.GlanceImageService
--network_host=controller
--vnc_console_proxy_url=http://controller:6080
--vncproxy_url=http://controller:6080
--vncproxy_host=controller
--vnc_enabled=False
--minimum_root_size=2147483648
--iscsi_ip_prefix=192.168.191
--libvirt_type=lxc
--use_cow_images=False

My DB version:
44

Tags: lxc

Related branches

Thierry Carrez (ttx)
tags: added: lxc
Thierry Carrez (ttx)
Changed in nova:
assignee: nobody → Chuck Short (zulcss)
importance: Undecided → High
status: New → In Progress
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2011.3
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.