LVM/DM/UDEV out of sync inside cinder-volumes container
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack-Ansible |
Fix Released
|
Critical
|
git-harry | ||
Icehouse |
Fix Released
|
Critical
|
git-harry | ||
Juno |
Fix Released
|
Critical
|
git-harry | ||
Kilo |
Fix Released
|
Critical
|
git-harry | ||
Trunk |
Fix Released
|
Critical
|
git-harry |
Bug Description
Apparently we can get into a situation that DM and UDEV are getting out of sync and causing all kinds of issues leading to inaccessibility of cinder-volumes LVM devices:
The /dev/mapper device is supposed to be a symlink but we turned off udev synchronization inside LXC containers to prevent CPU storm in udev situations. I this example, the device link points to the minor number 4
cinder_
brw-rw---- 1 root disk 252, 4 Mar 12 17:03 /dev/mapper/
BUT dm actually accesses this volume as minor number 6 :
cinder_
Name: cinder-
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 252, 6
Number of targets: 1
UUID: LVM-Ch2r8f3Sdrg
So after all it looks like we have to turn on UDEV sync (use_udev override) inside the cinder-volumes container.
Currently I do not know when this device ordering is happening, I suspect a creation and deletion of LVM volumes trigger a udev event which usually update the DM/LVM. I'll test this in a lab.
Changed in openstack-ansible: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
assignee: | nobody → Evan Callicoat (apsu-2) |
Changed in openstack-ansible: | |
milestone: | none → next |
Changed in openstack-ansible: | |
status: | Confirmed → In Progress |
I did some testing and was not able to reproduce this issue yet. device_ list_from_ udev or udev_rules did render LVM useless, even if udev manages /dev
"/dev" is not managed by udev inside the container unless you set a parameter but some parameters like obtain_
At some point I think not having /dev mounted over udev is causing more issues than it actually fixes. Ultimately I want to see udev working inside the container. We just have to check apparmor profiles etc or configurations to get the physical volume added for the cinder-volumes inside the container when using udev.