failed-to-delete-instance-lxc-with-lvm - filesystem in use
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
Description:
------------
When using lxc libvirt driver, nova encounters error in deleting an instance (nova delete). The instance is not deleted from nova. However, virsh list shows that the domain is not running after nova delete completes.
Steps to reproduce
--
libvirt_type=lxc to choose LXC as hypervisor type
libvirt_
nova boot container1
nova boot container2
nova delete container1
virsh -c lxc:/// list (to see what happens of the domain from point of view of libvirt).
Expected result
--
Nicely destroy container and remove LVM disk
Actual result
--
Nova encounters error in deleting container
Logs (n-cpu.log)
----
2016-08-08 16:06:08.500 ERROR oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
2016-08-08 16:06:08.500 TRACE oslo_messaging.
Related bug:
https:/
Environment:
-------------
Nova commit:
devdatta@
commit 4cbfed44f4b1dc4
Author: Anush Krishnamurthy <email address hidden>
Date: Wed Aug 3 05:06:06 2016 +0000
Update tags for Image file url from filesystems config option
Updated tags for config options consistency [1].
[1] https:/
Change-Id: I18bdca49ed8c8f
Implements: Blueprint centralize-
Hypervisor:
Libvirt + lxc
OS:
lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Kernel:
uname -a
Linux devdatta-
libvirt-bin:
dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13
lxc:
dpkg -l | grep lxc
ii liblxc1 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools (library)
ii lxc 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools
ii lxc-templates 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools (templates)
ii python3-lxc 1.0.8-0ubuntu0.3 amd64 Linux Containers userspace tools (Python 3.x bindings)
Storage type:
lvm
Network type:
nova-network
Some additional information:
1) Trying the command: rootwrap. conf lvremove -f /dev/stack- volumes- default/ 9fe6288f- e4e7-460f- 925d-9379ac0f74 fa_disk default/ 9fe6288f- e4e7-460f- 925d-9379ac0f74 fa_disk contains a filesystem in use.
sudo nova-rootwrap /etc/nova/
Logical volume stack-volumes-
2) Trying to unmount the volume: devdatta- nova-trusty- 2:/$ sudo umount /dev/stack- volumes- default/ 9fe6288f- e4e7-460f- 925d-9379ac0f74 fa_disk volumes- default/ 9fe6288f- e4e7-460f- 925d-9379ac0f74 fa_disk: not mounted
devdatta@
umount: /dev/stack-
3) Checking information about logical volumes: devdatta- nova-trusty- 2:/$ sudo lvs e4e7-460f- 925d-9379ac0f74 fa_disk stack-volumes- default -wi-ao--- 24.00m
devdatta@
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
9fe6288f-
*** The attributes (Attr) show that the logical volume is open (the 'o' in ao). ***
4) Checking the mapping of the volume devdatta- nova-trusty- 2:/$ ls -ltr /dev/stack- volumes- default/ 9fe6288f- e4e7-460f- 925d-9379ac0f74 fa_disk volumes- default/ 9fe6288f- e4e7-460f- 925d-9379ac0f74 fa_disk -> ../dm-2
devdatta@
lrwxrwxrwx 1 root root 7 Aug 8 16:06 /dev/stack-
5) Checking if any process is holding a reference to the volume devdatta- nova-trusty- 2:/$ sudo lsof | grep dm-2
devdatta@
jbd2/dm-2 11090 root cwd DIR 202,1 4096 2 /
jbd2/dm-2 11090 root rtd DIR 202,1 4096 2 /
jbd2/dm-2 11090 root txt unknown /proc/11090/exe
*** Looks like there is a process (11090) running that is holding handle to the volume. ***