OpenStack Compute (nova)

failed-to-delete-instance-lxc-with-lvm - filesystem in use

Bug #1611042 reported by Devdatta Kulkarni on 2016-08-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Confirmed	Medium	Unassigned

Bug Description

Description:
------------
When using lxc libvirt driver, nova encounters error in deleting an instance (nova delete). The instance is not deleted from nova. However, virsh list shows that the domain is not running after nova delete completes.

Steps to reproduce
--
libvirt_type=lxc to choose LXC as hypervisor type
libvirt_images_type=lvm to choose LVM as imagebackend store

nova boot container1
nova boot container2
nova delete container1
virsh -c lxc:/// list (to see what happens of the domain from point of view of libvirt).

Expected result
--
Nicely destroy container and remove LVM disk

Actual result
--
Nova encounters error in deleting container

Logs (n-cpu.log)
----

2016-08-08 16:06:08.500 ERROR oslo_messaging.rpc.server [req-d6b89fcc-9b02-4b75-a571-63bd0b82a336 demo demo] Exception during message handling
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server function_name, call_dict, binary)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 188, in decorated_function
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 613, in decorated_function
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2389, in terminate_instance
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server do_terminate_instance(instance, bdms)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2387, in do_terminate_instance
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self._set_instance_obj_error_state(context, instance)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2377, in do_terminate_instance
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self._delete_instance(context, instance, bdms, quotas)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/hooks.py", line 154, in inner
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server rv = f(*args, **kwargs)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2340, in _delete_instance
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server quotas.rollback()
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2196, in _shutdown_instance
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server block_device_info)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 923, in destroy
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server destroy_disks, migrate_data)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1019, in cleanup
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server self._cleanup_lvm(instance, block_device_info)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1085, in _cleanup_lvm
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server lvm.remove_volumes(disks)
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/storage/lvm.py", line 241, in remove_volumes
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server raise exception.VolumesNotRemoved(reason=(', ').join(errors))
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server VolumesNotRemoved: Failed to remove volume(s): (Unexpected error while running command.
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server Command: sudo nova-rootwrap /etc/nova/rootwrap.conf lvremove -f /dev/stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server Exit code: 5
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server Stdout: u''
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server Stderr: u' Logical volume stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk contains a filesystem in use.\n')
2016-08-08 16:06:08.500 TRACE oslo_messaging.rpc.server

Related bug:
https://bugs.launchpad.net/nova/+bug/1091639

Environment:
-------------

Nova commit:
devdatta@devdatta-nova-trusty-2:/opt/stack/nova$ git log -1
commit 4cbfed44f4b1dc4af21e9a09ed830617cb3caa05
Author: Anush Krishnamurthy <email address hidden>
Date: Wed Aug 3 05:06:06 2016 +0000

Update tags for Image file url from filesystems config option

Updated tags for config options consistency [1].

[1] https://wiki.openstack.org/wiki/ConfigOptionsConsistency

Change-Id: I18bdca49ed8c8f5fb0bc6782526dca14bd5c8646
Implements: Blueprint centralize-config-options-newton

Hypervisor:
Libvirt + lxc

OS:
lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04

Kernel:
uname -a
Linux devdatta-nova-trusty-2 3.13.0-88-generic #135-Ubuntu SMP Wed Jun 8 21:10:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

libvirt-bin:
dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.17 amd64 programs for the libvirt library

lxc:
dpkg -l | grep lxc
ii liblxc1 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools (library)
ii lxc 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools
ii lxc-templates 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools (templates)
ii python3-lxc 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools (Python 3.x bindings)

Storage type:
lvm

Network type:
nova-network

Tags:

Revision history for this message

Devdatta Kulkarni (devdatta-kulkarni) wrote on 2016-08-08:

Some additional information:

1) Trying the command:
sudo nova-rootwrap /etc/nova/rootwrap.conf lvremove -f /dev/stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk
Logical volume stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk contains a filesystem in use.

2) Trying to unmount the volume:
devdatta@devdatta-nova-trusty-2:/$ sudo umount /dev/stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk
umount: /dev/stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk: not mounted

3) Checking information about logical volumes:
devdatta@devdatta-nova-trusty-2:/$ sudo lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk stack-volumes-default -wi-ao--- 24.00m

*** The attributes (Attr) show that the logical volume is open (the 'o' in ao). ***

4) Checking the mapping of the volume
devdatta@devdatta-nova-trusty-2:/$ ls -ltr /dev/stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk
lrwxrwxrwx 1 root root 7 Aug 8 16:06 /dev/stack-volumes-default/9fe6288f-e4e7-460f-925d-9379ac0f74fa_disk -> ../dm-2

5) Checking if any process is holding a reference to the volume
devdatta@devdatta-nova-trusty-2:/$ sudo lsof | grep dm-2
jbd2/dm-2 11090 root cwd DIR 202,1 4096 2 /
jbd2/dm-2 11090 root rtd DIR 202,1 4096 2 /
jbd2/dm-2 11090 root txt unknown /proc/11090/exe

*** Looks like there is a process (11090) running that is holding handle to the volume. ***

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-09-09:

Yeah this is a long-standing issue and is the main blocker from us getting the LXC CI job working, is we have something locking here so we can't delete the instances.

Can you figure out what that 11090 process is?

Changed in nova:
status:	New → Confirmed
importance:	Undecided → Medium
summary:	- failed-to-delete-instance-lxc-with-lvm + failed-to-delete-instance-lxc-with-lvm - filesystem in use

Revision history for this message

Sean Dague (sdague) wrote on 2017-06-27:

Automatically discovered version newton in description. If this is incorrect, please update the description to include 'nova version: ...'

tags:

added: openstack-version.newton

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.