udev breaks multipath after reboot when using LVM

[cuicui]

Binary package hint: udev

Hello,

 * Summary: I'm using an EMC Clariion CX3-10 san controller on a Dell PowerEdge M100 server., Ubuntu Server "Hardy" 8.04, amd-64 as operating System. Once the "multipath-tool" package is installed, a new device is created in /dev/mapper, after creating a "0x8E" type partition on it and creating a PV / VG / LG with LVM on this partition I have a fonctionnal LVM Logical Group working with multipath. After a reboot, the multipath configuration is lost (no failover / load-balancing).

 * Step to reproduce:

 - Having Ubuntu Hardy Server 8.04-amd-64 installed with packages "lvm2" and "multipath-tools"
 - Create /etc/multipath.conf with and alias of the WWID for easy reading and restart the multipath-tools service

==========================================================

# cat /etc/multipath.conf

multipaths {
       multipath {
               wwid 36006016071202000c4f6e817ea20dd11
               alias lun3
       }
}

==========================================================

 - Check the output of "multipath -ll" ("lun3" is the alias)

==========================================================

# multipath -ll
lun3 (36006016071202000c4f6e817ea20dd11) dm-0 DGC ,RAID 5
[size=50G][features=1 queue_if_no_path][hwhandler=1 emc]
\_ round-robin 0 [prio=2][active]
 \_ 3:0:1:0 sdc 8:32 [active][ready]
 \_ 4:0:1:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 3:0:0:0 sda 8:0 [active][ready]
 \_ 4:0:0:0 sdd 8:48 [active][ready]

==========================================================

 - Check the presence of "/dev/mapper/lun3"
 - Use "fdisk /dev/mapper/lun3" and create a new primary partition, change the partition's type to "0x8E" (LVM)
 - Restart the "udev" service
 - Check the presence of "/dev/mapper/lun3-part1"
 - Create a PV with "pvcreate /dev/mapper/lun3-part1"
 - Add the line "types = [ "device-mapper", 1 ]" in the /etc/lvm.conf file
 - Add this PV on a new VG with "vgcreate vg0 /dev/mapper/lun3-part1"
 - Create a new LV on this VG with "lvcreate -l 12799 -n lv0 vg0"
 - Create a new filesystem on this LV with "mkfs.ext3 /dev/vg0/lv0"
 - Check the output of "multipath -ll", it should be the same as before
 - Check the output of "pvdisplay"

==========================================================

# pvdisplay
  --- Physical volume ---
  PV Name /dev/mapper/lun3-part1
  VG Name vg0
  PV Size 50,00 GB / not usable 3,28 MB
  Allocatable yes (but full)
  PE Size (KByte) 4096
  Total PE 12799
  Free PE 0
  Allocated PE 12799
  PV UUID R3lJLm-ZTUH-KuBg-UwIM-HhmP-1iWn-yWrGeH

==========================================================

 - At this point both multipating and LVM are fonctional
 - reboot the server

 * Actual result:

 - After the reboot, the "multipath -ll" command isn't giving any output => multipathing is not functional anymore.
 - The /dev/mapper/lun3 and /dev/mapper/lun3-part1 files aren't there anymore
 - The output of "pvdisplay" show that the "PV name" isn't the same as before:

==========================================================

# pvdisplay
  --- Physical volume ---
  PV Name /dev/disk/by-id/scsi-36006016071202000c4f6e817ea20dd11-part1
  VG Name vg0
  PV Size 50,00 GB / not usable 3,28 MB
  Allocatable yes (but full)
  PE Size (KByte) 4096
  Total PE 12799
  Free PE 0
  Allocated PE 12799
  PV UUID R3lJLm-ZTUH-KuBg-UwIM-HhmP-1iWn-yWrGeH

==========================================================

 * Expected result:

 - Multipathing should work
 - The /dev/mapper/lun3 and /dev/mapper/lun3-part3 files should exists
 - It looks like a mixup between the udev rules for LVM / dmsetup / multipath / kpartx

 * Note:

 - When removing all the LVM configuration with "lvremove", "vgremove", "pvremove" and restarting the "multipath-tools" service, the "/dev/mapper/lun3" and "/dev/mapper/lun3-part1" are back again. The "multipath -ll" output is also back to normal => multipathing is running fine.

 - This problem doesn't seems to affect others distrib, a RHEL How-To explains the use of Multipathing + LVM, see: https://secure.linux.ncsu.edu/moin/RealmLinuxServers/Multipathing

 - The official maintainer of the "multipath-package" talks about the use of LVM and the udev rules in the official FAQ, see: http://christophe.varoqui.free.fr/faq.html

[Ken Simon (ninkendo)]

This bug's gone pretty much unnoticed I guess, but I'm experiencing the same issue. I'm scheduling downtime for next thursday (unfortunately this is a production machine) to try out some different LVM configurations, to make sure I can get it to reliably reboot with LVM scanning the multipath nodes, instead of the raw (sd*) device nodes.

Here's the configuration I'm using and expecting to work:

lvm.conf:

devices {
        dir="/dev"
        scan="/dev"
        filter=["a|/dev/mapper/mpath.*|", "r|.*|"]
        cache="/etc/lvm/.cache"
        write_cache_state=1
        types=["device-mapper", 1]
        sysfs_scan=1
        md_component_detection=1
}

Note that "types=" contains device-mapper types. IIRC this is not by default in Ubuntu, but is required for multipath support.

Also note that "filter=" is telling LVM to only scan for /dev/mapper/mpath*. This will make sure that LVM is only scanning for devices in /dev/mapper that begin with mpath. To get multipath to make devices with this prefix, edit multipath.conf:

defaults {
        ...
        user_friendly_names yes #names the devices mpath0 - mpath<n>
        ...
}

Or, optionally configure aliases for each multipath, and have them begin with "mpath".

These changes *should* force LVM to only ever look at /dev/mapper, and for multipath devices in particular, for PV's. If you use LVM for more than just mpio, change the filter line in lvm.conf to something like:

filter=["a|/dev/mapper/mpath.*|", "a|^/dev/sda1$|", "a|^/dev/sdb1$|", "r|.*|"]

This will add /dev/sda1 and /dev/sdb1 in addition to the mpio devices.

I'll update this bug next thursday to say whether it works or not after a reboot.

[cuicui (cuicui-oizo)]

I tried your settings, this is before reboot:

$ sudo pvdisplay
  --- Physical volume ---
  PV Name /dev/mapper/mpath1-part1
  VG Name vg0
  PV Size 2,00 TB / not usable 1,23 MB
  Allocatable yes (but full)
  PE Size (KByte) 4096
  Total PE 524287
  Free PE 0
  Allocated PE 524287
  PV UUID znfa6r-X7FD-VXTd-WdsJ-LAu2-8SdS-relheA

I created a PV and a LV: /dev/vg0/lv0, /dev/mapper/mpath1-part1 is present.

After reboot, /dev/vg0/lv0 is still there, but /dev/mapper/mpath1-part1 has disappeared and /dev/mapper/vg0-lv0 is showing up.

"pvdisplay" and "multipath -ll" give no output but /dev/vg0/lv0 can be mounted (probably using only one static path)

I used your "devices" part in /dev/lvm/lvm.conf and my /etc/multipath.conf contains only "defaults { user_friendly_names yes }".

If you want to run more tests, go ahead, I'm using a sandbox blade server.

[Ken Simon (ninkendo)]

Hmm, that's odd indeed. Let's disect this.

pvdisplay is showing nothing, because it's reading the config file, and only looking for physical volumes in /dev/mapper, in which there are none. (Well, none other than LVM's volume itself.)

multipath -ll is showing nothing, because it never created any multipaths, because LVM is already locking and using the static paths. (At least, that's my guess.)

So the mystery here is, what's happening at boot that's letting LVM see the raw (static) paths to the devices, and ignoring lvm.conf's instructions to only look in /dev/mapper?

My guess is it's something in ubuntu's initramfs, where LVM has a default config file that lets it look wherever it wants for PV's, and LVM ends up picking the raw (non-multipath) devices, instead of waiting for multipathd to come in and create multipath nodes in /dev/mapper. This is probably necessary for the kernel to boot from an LVM device... after all, you can't read /etc/lvm/lvm.conf if / isn't mounted. (and / could be on LVM.)

I'm going to poke around ubuntu's initramfs and see if there's something I can find there. Would it be alright if I had you set up a new initramfs, and try booting from it?

[Ken Simon (ninkendo)]

Ooh, I found a great example of what could be causing this:

https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/113515

Basically, the bug is that, since /etc/lvm/lvm.conf is *not* in your initramfs, LVM defaults to scanning all devices for PV's, and locks onto the first ones it sees (which it shouldn't.)

I'm not sure whether the above bug was fixed yet or not, though. Make sure you have the changes I suggested for lvm.conf, then try updating your initramfs with:

update-initramfs -c -k `uname -r`

then reboot and try again.

[cuicui (cuicui-oizo)]

Rebuilding the initramfs doesn't help, but including a new hook for LVM first solve the problem:

 - sudo cp /usr/share/initramfs-tools/hooks/lvm2 /etc/initramfs-tools/hooks/
 - sudo update-initramfs -c -k `uname -r`

Works also with aliases in /etc/multipath.conf.

Unless you have something to had, I think this bug can be closed.

Regards,

[Ken Simon (ninkendo)]

To be clear, did you do anything to /etc/initramfs-tools/hooks before you built the ramdisk? Because I was about to post again to point out that you should add:

if [ -e /etc/lvm/lvm.conf ]; then
    mkdir -p ${DESTDIR}/etc/lvm
    cp /etc/lvm/lvm.conf ${DESTDIR}/etc/lvm/
fi

to /usr/share/initramfs-tools/hooks/lvm2, before building your initramfs. In fact, I'm pretty sure update-initramfs will scan both /usr/shares/initramfs-tools/hooks and /etc/initramfs-tools/hooks when it's creating the ramdisk.

Anyways, I made the above change, checked the resulting ramdisk with cpio, and sure enough, this change produced an initramfs that contains an /etc/lvm/lvm.conf.

The second step I took was to install the package multipath-tools-boot, which contains scripts to initialize multipath on bootup, (hopefully) before LVM runs.

Could you make these changes, and make sure that LVM consistently mounts the proper device nodes (/dev/mapper/mpath*) on every boot?

Much appreciated.

[cuicui (cuicui-oizo)]

At this time, I have a functional system, but /etc/lvm/lvm.conf is not included in my initramfs. I did not edit any file besides /etc/lvm/lvm.conf to filter out devices. This is the diff between a working (good) and a not working (bad) initramfs:

=====================================

$ LANG=en diff -r good bad
Binary files good/bin/dmesg and bad/bin/dmesg differ
Only in good/bin: ntfs-3g
diff -r good/conf/conf.d/resume bad/conf/conf.d/resume
1c1
< RESUME=UUID=2a90babe-f5f4-4954-8844-9f069a8b8abf
---
> RESUME=/dev/sdb3
diff -r good/conf/modules bad/conf/modules
1d0
< fuse
7,9d5
< dm_mod
< dm_snapshot
< dm_mirror
Binary files good/etc/console-setup/boottime.kmap.gz and bad/etc/console-setup/boottime.kmap.gz differ
diff -r good/etc/default/console-setup bad/etc/default/console-setup
8c8
< BOOTTIME_KMAP_MD5="4c25c2e93531789eeb1644f36883c96d"
---
> BOOTTIME_KMAP_MD5="24de09e93760848ccb47b26a64b2e66a"
Only in good/etc/modprobe.d: fuse
Only in good/etc/modprobe.d: lrm-video
Only in bad/etc/modprobe.d: lrm-video.dpkg-new
Only in good/etc/modprobe.d: nvidia-kernel-nkc
Only in bad/etc/modprobe.d: nvidia-kernel-nkc.dpkg-new
Only in good/etc/udev/rules.d: 65-dmsetup.rules
Only in good/etc/udev/rules.d: 85-lvm2.rules
Binary files good/lib/libc.so.6 and bad/lib/libc.so.6 differ
Only in good/lib: libdevmapper.so.1.02.1
Only in good/lib: libfuse.so.2
Only in good/lib: libncurses.so.5
Only in good/lib: libntfs-3g.so.23
Binary files good/lib/libproc-3.2.7.so and bad/lib/libproc-3.2.7.so differ
Only in good/lib: libpthread.so.0
Only in good/lib: libreadline.so.5
Only in good/lib: librt.so.1
Only in good/lib: libsepol.so.1
Only in good/lib/modules/2.6.24-23-generic/kernel/drivers: md
Only in good/lib/modules/2.6.24-23-generic/kernel/fs: fuse
Binary files good/sbin/depmod and bad/sbin/depmod differ
Only in good/sbin: dmsetup
Only in good/sbin: lvm
Binary files good/sbin/modprobe and bad/sbin/modprobe differ
Only in good/sbin: mount.fuse
Only in good/sbin: mount.ntfs
Only in good/sbin: mount.ntfs-3g
Binary files good/sbin/pkill and bad/sbin/pkill differ
Binary files good/sbin/rmmod and bad/sbin/rmmod differ
Only in good/scripts/init-premount: lvm2
diff -r good/scripts/init-top/console_setup bad/scripts/init-top/console_setup
27,30d26
< for i in 1 2 3 4 5 6; do
< [ -c /dev/tty$i ] || mknod /dev/tty$i c 4 $i
< done
<
Only in good/scripts/local-bottom: ntfs_3g
Only in good/scripts/local-premount: ntfs_3g

=====================================

regarding lvm:

Only in good/etc/udev/rules.d: 85-lvm2.rules
Only in good/sbin: lvm
Only in good/scripts/init-premount: lvm2

=====================================

There's something I don't understand about how initramfs is updated: it seems that you're right and that the lvm2 hook is called even if it's not present in /etc/initramfs/hooks/.

I'll try to make a fresh install and a backup and try to reproduce the behavior. In my point of view, everything worked out the moment I copied /usr/share/initramfs-tools/hooks/lvm2 inito /etc/initramfs-tools/hooks/.

Now I can remove /etc/initramfs-tools/hooks/lvm2 and update initramfs: it's still working.

Maybe update-initramfs includes the lvm2 hook (present in /etc/initramfs-too...

[cuicui (cuicui-oizo)]

I confirm that a simple initramfs update with no explicit include of /etc/lvm/lvm.conf or any other modification works. The LVM is correctly mounted at boot time.