Cinder with LVM backend scans guest LVM and creates LVM items which prevent volume deletion

[Maxim V. Yefimov]

Environment:

MOS7.0
MOS8.0
Cinder with LVM/iSCSI backend

Step to reproduce:

1. Create volume;
2. Attach to an instance;
3. Create PV, VG and LV on this volume inside the instance;
4. Shutdown instance or detach volume;
5. Delete volume;

Expected result: Deleted volume

Actual result: Volume gets stuck on error_deleting state

Cause: Device mapper on the storage node created the following structure:

# dmsetup ls --tree
vg--01-vol--01 (252:4)
 └─cinder-volume--3e6737c9--6306--4d93--9af1--85bfdd6a5f9a (252:3)
    └─ (253:5)
vg-01 - guest VG
vol-01 - guest LV

As result, the Cinder volume is dependent on this item and can't be deleted.

Workaround: Set global_filter parameter in the /etc/lvm/lvm.conf file on all Storage nodes to ["r/dev/sd*./"]

[Ivan Kolodyazhny (e0ne)]

It's not a Cinder issue. We have to change /etc/lvm/lvm.conf during deployment

[Nikita Karpin (mkarpin)]

I was able to reproduce this on 9.x.

I'd like to notice that changing of global_filter option during deployment is more a feature than a bug fix, because it is very user-specific:

- local disks on compute could be named in different way (sd*, vd*, hd* and so on)

- it is impossible to predict which devices end-user wants to mount to the compute node - what if he wants to mount some devices with filtered name which contains LVs, PVs inside?

So in order to fix this we need at least investigate how to differentiate volumes mounted for guests from other volumes and as better solution we need also to make the global_filter configurable by end-user.

So I would suggest to describe this as known issue in documentation and describe a workaround for it.

[Marcus Furlong (furlongm)]

Isn't it possible to reverse filter? I.e. just to exclude certain devices. In that case, if we know the cinder volumes will be in a VG called 'cinder' or 'cinder-volumes', we can exclude /dev/cinder/* or /dev/cinder-volumes/*.

[Nikita Karpin (mkarpin)]

Thanks Marcus, I looked further at it and found that this filter works:

global_filter = [ "r/dev/cinder/.*/" ]

after this we should do 'update-initramfs -u' in order to ensure that changes are kept during reboot.

So in this way we don't need to care about names of host's disks, we need just know name of Cinder volume group which is created by Fuel during provisioning

[Nikita Karpin (mkarpin)]

https://review.openstack.org/#/c/411759/ https://review.openstack.org/#/c/411754/ fixes are merged

[TatyanaGladysheva (tgladysheva)]

Issue is still reproduced on 9.2 snapshot #685.
Move to Confirmed status.

[Nikita Karpin (mkarpin)]

I found that augeas 1.2 installed on Ubuntu Trusty does not support parsing of lvm.conf, so looks like I need to use file_line type in puppet manifest, will prepare new fix for 9.x

[Nikita Karpin (mkarpin)]

correct fix for 9.x - https://review.openstack.org/#/c/415437/

[Dmitry Pyzhov (dpyzhov)]

Fix merged and tested for 9.2

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #724.

Actual results:
http://paste.openstack.org/show/594359/

[Sergii Rizvan (srizvan)]

Here are the patches for
7.0: https://review.openstack.org/#/c/422768/
8.0: https://review.openstack.org/#/c/421362

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 7.0 + MU7 updates.

Actual results:
http://paste.openstack.org/show/599636/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 8.0 + MU4 updates.

Actual results:
http://paste.openstack.org/show/600760/

[TatyanaGladysheva (tgladysheva)]

Verified on 10.0 build #1569.

[Margarita Shakhova (shakhova-margarita)]

Similar problem was observed for EMC VNX backend.