initramfs boot hang (waiting for termination ofwatershed vgscan vgchange proccesses) if logical volume contains volume group

[Andreas Greve]

Hello,

I have a problem like in

   https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/802626
   vgchange may deadlock in initramfs when VG present that's not used for rootfs

but I'm not sure.

If there exist a logical volume (/dev/mapper/nova--volumes-nova--disk /dev/dm-1) which self contains a volume group (XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686) which contains a logical volume (XSLocalEXT--5c88197f--6ec8--9a68--82f3--a475be0cd686-5c88197f--6ec8--9a68--82f3--a475be0cd686 /dev/dm-23) the system will drop to initramfs shell on booting.

The reason is the that the wateshed processes (lvm lvscan; lvm vgchange -a -y ) hangs until they are killed by timeout but at this point the boot process is already droped to initramfs shell. After the kill of the watershed processes all looks well , all logical volumes found and symlinks are constructed to /dev/dm-*. Leaving the initramfs shell is the only thing to do to continue the boot process.

To analyse the problem and as work around I have modified the initramfs-tools files scripts/function and scripts/local

in scripts/functions I add a function ls8panic

ls8panic() {
 if command -v chvt >/dev/null 2>&1; then
  chvt 1
 fi
        echo "ls8panic: entered reason: $@"
        ls -l /dev/mapper
        lvm pvs
        lvm lvs
        ps | grep -v grep | grep watershed
        if [ $? -eq 0 ]; then
          #watershed is running
          echo "lspanic: wateshed is running sleep 15 secs than do as handled and continue."
          sleep 15
         else
          #watershed is not running
          echo "ls8panic: watershed not running sleep 1 secs do a 'vgscan && vgschange -a y' than do as handled and continue."
          sleep 1
          lvm vgscan && lvm vgchange -a y
        fi
        ls -l /dev/mapper
        lvm pvs
        lvm lvs
        echo "ls8panic: leaved."
}

and in scripts/local I modified the function pre_mountroot() so that the routine ls8panic() was called 6 times before the function panic() will be called:

pre_mountroot()
{
        [...]
 done
        _ls8count="654321"
 # We've given up, but we'll let the user fix matters if they can
 while [ -z "${FSTYPE}" -a ! -e "${ROOT}" ]; do
        [...]
  esac
                if [ -n "${_ls8count}" ]; then
                  _ls8count="${_ls8count%?}"
                  ls8panic "ALERT!(${_ls8count}) ${ROOT} does not exist."
                else
  echo "Gave up waiting for root device. Common problems:"
                [...]
}

What you can see in the second example is that the watershed proccesses seams to hang after creating the symlinks for the volume which contains a volume group.

lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1

but after the termination of the watershed proccesses all expected symlinks in /dev/mapper created.

A second dubios thin are the messages

    udevd[467]: inotify_add_watch(6, /dev/dm-22, 10) failed: Invalid argument
    udevd[476]: inotify_add_watch(6, /dev/dm-23, 10) failed: Invalid argument

in the working boot and

    udevd[138]: inotify_add_watch(6, /dev/dm-24, 10) failed: No such file or directory

in the "hanging boot.

I dont know nothing about wath volumes/devices dm-22, dm-23 or dm-24 should be or where they come from.

I hope the information will help.

Thanks

Andreas

Version numbers of used packages:

initramfs-tools 0.99ubuntu13
udev 175-0ubuntu9.1
lvm2 2.02.66-4ubuntu7.1
watershed 6

Partion and pv configuration of used disks:
root@ls8vmw04:~# fdisk -l /dev/sda /dev/sdb

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00078fab

   Device Boot Start End Blocks Id System
/dev/sda1 * 63 835379 417658+ 83 Linux
/dev/sda4 835380 488392064 243778342+ 5 Extended
/dev/sda5 835443 488392064 243778311 8e Linux LVM

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x936e4c69

   Device Boot Start End Blocks Id System
/dev/sdb1 63 1012094 506016 83 Linux
/dev/sdb2 1012095 1953520064 976253985 5 Extended
/dev/sdb5 1012158 525306557 262147200 8e Linux LVM
/dev/sdb6 525308606 1049596605 262144000 8e Linux LVM
/dev/sdb7 1049598654 1469029053 209715200 8e Linux LVM

root@ls8vmw04:~# pvs
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0

interesting part of initramfs messages copied from console for a working boot:

[...]
[ 2.134160] usbhid: USB HID core driver
[ 2.153411] ata3.00: configured for UDMA/133
[ 2.155647] scsi 2:0:0:0: Direct-Access ATA SEAGATE ST32500N n/a PQ: 0 ANSI: 5
[ 2.159954] sd 2:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 2.164171] sd 2:0:0:0: [sda] Write Protect is off
[ 2.166332] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 2.166423] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.197409] sda: sda1 sda4 < sda5 >
[ 2.200043] sd 2:0:0:0: [sda] Attached SCSI disk
[ 2.368070] usb 2-5: new low-speed USB device number 3 using ohci_hcd
[ 2.599394] input: HID 046a:0011 as /devices/pci0000:00/0000:00:02.0/usb2/2-5/2-5:1.0/input/input4
[ 2.603987] generic-usb 0003:046A:0011.0003: input,hidraw2: USB HID v1.11 Keyboard [HID 046a:0011] on usb-0000:00:02.0-5/input0
[ 2.636905] forcedeth 0000:00:09.0: ifname eth3, PHY OUI 0x5043 @ 3, addr 00:1e:68:04:a6:fe
[ 2.640091] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2.643184] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[ 2.649015] ata4.00: ATA-8: WDC WD1003FBYX-01Y7B1, 01.01V02, max UDMA/133
[ 2.651139] ata4.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[ 2.661025] ata4.00: configured for UDMA/133
[ 2.663266] scsi 3:0:0:0: Direct-Access ATA WDC WD1003FBYX-0 01.0 PQ: 0 ANSI: 5
[ 2.667792] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 2.667812] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 2.674331] sd 3:0:0:0: [sdb] Write Protect is off
[ 2.676517] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.729049] sdb: sdb1 sdb2 < sdb5 sdb6 sdb7 >
[ 2.731724] sd 3:0:0:0: [sdb] Attached SCSI disk
Begin: Running /scripts/local-premount ... done.
[ 5.376959] EXT3-fs (dm-20): recovery required on readonly filesystem
[ 5.379085] EXT3-fs (dm-20): write access will be enabled during recovery
[ 5.886755] kjournald starting. Commit interval 5 seconds
[ 5.886926] EXT3-fs (dm-20): recovery complete
[ 5.906886] EXT3-fs (dm-20): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... udevd[467]: inotify_add_watch(6, /dev/dm-22, 10) failed: Invalid argument

udevd[476]: inotify_add_watch(6, /dev/dm-23, 10) failed: Invalid argument

done.
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
/dev/mapper/vg00-ls8vmw04_root: clean, 77091/1048576 files, 680568/4194304 blocks
/dev/sda1: clean, 298/102408 files, 183459/417656 blocks
rpcbind: Cannot open '/run/rpcbind/rpcbind.xdr' file for reading, errno 2 (No such file or directory)
rpcbind: Cannot open '/run/rpcbind/portmap.xdr' file for reading, errno 2 (No such file or directory)
[...]

interesting part of initramfs messages copied from console for a failed boot:

[ 2.100264] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 23 (level, low) -> IRQ 23
[ 2.105384] ata1.00: configured for UDMA/133
[ 2.107606] scsi 0:0:0:0: Direct-Access ATA SEAGATE ST32500N n/a PQ: 0 ANSI: 5
[ 2.111950] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 2.112009] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 2.112070] sd 0:0:0:0: [sda] Write Protect is off
[ 2.112107] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.149403] sda: sda1 sda4 < sda5 >
[ 2.151894] sd 0:0:0:0: [sda] Attached SCSI disk
[ 2.328074] usb 2-5: new low-speed USB device number 3 using ohci_hcd
[ 2.560399] input: HID 046a:0011 as /devices/pci0000:00/0000:00:02.0/usb2/2-5/2-5:1.0/input/input4
[ 2.564809] generic-usb 0003:046A:0011.0003: input,hidraw2: USB HID v1.11 Keyboard [HID 046a:0011] on usb-0000:00:02.0-5/input0
[ 2.592090] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2.601265] ata2.00: ATA-8: WDC WD1003FBYX-01Y7B1, 01.01V02, max UDMA/133
[ 2.603396] ata2.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[ 2.613287] ata2.00: configured for UDMA/133
[ 2.615531] scsi 1:0:0:0: Direct-Access ATA WDC WD1003FBYX-0 01.0 PQ: 0 ANSI: 5
[ 2.619858] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 2.619880] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 2.626404] sd 1:0:0:0: [sdb] Write Protect is off
[ 2.626785] scsi 2:0:0:0: CD-ROM TEAC DV-28E-V 1.AC PQ: 0 ANSI: 5
[ 2.628861] forcedeth 0000:00:09.0: ifname eth3, PHY OUI 0x5043 @ 3, addr 00:1e:68:04:a6:fe
[ 2.628864] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[ 2.635406] sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[ 2.635408] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 2.635713] sr 2:0:0:0: Attached scsi generic sg2 type 5
[ 2.648098] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2.709774] sdb: sdb1 sdb2 < sdb5 sdb6 sdb7 >
[ 2.712393] sd 1:0:0:0: [sdb] Attached SCSI disk
ls8panic: entered reason: ALERT!(65432) /dev/mapper/vg00-ls8vmw04_root does not exist.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
  286 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  289 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  291 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  292 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  304 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
lspanic: wateshed is running sleep 15 secs than do as handled and continue.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
ls8panic: leaved.
ls8panic: entered reason: ALERT!(6543) /dev/mapper/vg00-ls8vmw04_root does not exist.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
  286 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  289 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  291 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  292 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  304 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
lspanic: wateshed is running sleep 15 secs than do as handled and continue.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
ls8panic: leaved.
ls8panic: entered reason: ALERT!(654) /dev/mapper/vg00-ls8vmw04_root does not exist.
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi--- 29.99g
  nova-disk nova-volumes -wi-a- 30.00g
  dom0_mysql vg00 -wi--- 10.00g
  dom0_postgresql vg00 -wi--- 10.00g
  dom0_root vg00 -wi--- 16.00g
  dom0_root_ULTS12 vg00 -wi--- 16.00g
  dom0_swap vg00 -wi--- 32.00g
  ls8pc09_root vg00 -wi--- 16.00g
  ls8pc09_swap vg00 -wi--- 16.00g
  ls8vmw04_root vg00 -wi--- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi--- 4.00g
  ls8vmw04_root_old vg00 -wi--- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi--- 4.00g
  ls8vmw04_swap vg00 -wi--- 2.00g
  dom0_root_LTS12 vg01 -wi--- 16.00g
  dom0_root_LTS8 vg01 -wi--- 16.00g
  dom0_swap vg01 -wi--- 4.00g
  kilab065-disk vg01 -wi--- 16.00g
  kilab065-hadoop vg01 -wi--- 100.00g
  kilab065-swap vg01 -wi--- 8.00g
  ls8olc04-disk vg01 -wi--- 16.00g
  ls8olc04-swap vg01 -wi--- 16.00g
  lvswift vgswift -wi-a- 200.00g
  286 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  289 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  291 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  292 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
  304 0 4300 S /lib/udev/watershed sh -c /sbin/lvm vgscan; /sbin/lv
lspanic: wateshed is running sleep 15 secs than do as handled and continue.
udevd[140]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [292]

udevd[142]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [289]

udevd[138]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [286]

udevd[140]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [292] terminated by signal 9 (Killed)

udevd[138]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [286] terminated by signal 9 (Killed)

udevd[142]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [289] terminated by signal 9 (Killed)

udevd[146]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [291]

udevd[146]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [291] terminated by signal 9 (Killed)

udevd[137]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [304]

udevd[137]: 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' [304] terminated by signal 9 (Killed)

udevd[138]: inotify_add_watch(6, /dev/dm-24, 10) failed: No such file or directory

crw------- 1 10, 236 control
lrwxrwxrwx 1 7 vgswift-lvswift -> ../dm-0
lrwxrwxrwx 1 7 nova--volumes-nova--disk -> ../dm-1
lrwxrwxrwx 1 7 vg01-kilab065--hadoop -> ../dm-2
lrwxrwxrwx 1 7 vg01-ls8olc04--swap -> ../dm-3
lrwxrwxrwx 1 7 vg01-dom0_swap -> ../dm-4
lrwxrwxrwx 1 7 vg01-dom0_root_LTS8 -> ../dm-6
lrwxrwxrwx 1 7 vg01-kilab065--swap -> ../dm-5
lrwxrwxrwx 1 7 vg01-dom0_root_LTS12 -> ../dm-7
lrwxrwxrwx 1 7 vg01-ls8olc04--disk -> ../dm-9
lrwxrwxrwx 1 7 vg01-kilab065--disk -> ../dm-8
lrwxrwxrwx 1 8 vg00-dom0_root -> ../dm-10
lrwxrwxrwx 1 8 vg00-ls8pc09_root -> ../dm-12
lrwxrwxrwx 1 8 vg00-dom0_swap -> ../dm-11
lrwxrwxrwx 1 8 vg00-dom0_root_ULTS12 -> ../dm-14
lrwxrwxrwx 1 8 vg00-dom0_mysql -> ../dm-15
lrwxrwxrwx 1 8 vg00-dom0_postgresql -> ../dm-16
lrwxrwxrwx 1 8 vg00-ls8pc09_swap -> ../dm-13
lrwxrwxrwx 1 8 vg00-ls8vmw04_swap -> ../dm-17
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_back_base_ohne_openstack -> ../dm-19
lrwxrwxrwx 1 8 vg00-ls8vmw04_root -> ../dm-20
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack -> ../dm-21
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_old -> ../dm-18
lrwxrwxrwx 1 8 vg00-ls8vmw04_root_with_xcp_partiell_openstack -> ../dm-22
lrwxrwxrwx 1 8 XSLocalEXT--5c88197f--6ec8--9a68--82f3--a475be0cd686-5c88197f--6ec8--9a68--82f3--a475be0cd686 -> ../dm-23
  PV VG Fmt Attr PSize PFree
  /dev/dm-1 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 lvm2 a- 29.99g 0
  /dev/sda5 vg00 lvm2 a- 232.48g 70.48g
  /dev/sdb5 vg01 lvm2 a- 250.00g 58.00g
  /dev/sdb6 nova-volumes lvm2 a- 250.00g 220.00g
  /dev/sdb7 vgswift lvm2 a- 200.00g 0
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  5c88197f-6ec8-9a68-82f3-a475be0cd686 XSLocalEXT-5c88197f-6ec8-9a68-82f3-a475be0cd686 -wi-a- 29.99g
  nova-disk nova-volumes -wi-ao 30.00g
  dom0_mysql vg00 -wi-a- 10.00g
  dom0_postgresql vg00 -wi-a- 10.00g
  dom0_root vg00 -wi-a- 16.00g
  dom0_root_ULTS12 vg00 -wi-a- 16.00g
  dom0_swap vg00 -wi-a- 32.00g
  ls8pc09_root vg00 -wi-a- 16.00g
  ls8pc09_swap vg00 -wi-a- 16.00g
  ls8vmw04_root vg00 -wi-a- 16.00g
  ls8vmw04_root_back_base_ohne_openstack vg00 -wi-a- 4.00g
  ls8vmw04_root_back_base_with_openswitch_xcp_ohne_openstack vg00 -wi-a- 4.00g
  ls8vmw04_root_old vg00 -wi-a- 16.00g
  ls8vmw04_root_with_xcp_partiell_openstack vg00 -wi-a- 4.00g
  ls8vmw04_swap vg00 -wi-a- 2.00g
  dom0_root_LTS12 vg01 -wi-a- 16.00g
  dom0_root_LTS8 vg01 -wi-a- 16.00g
  dom0_swap vg01 -wi-a- 4.00g
  kilab065-disk vg01 -wi-a- 16.00g
  kilab065-hadoop vg01 -wi-a- 100.00g
  kilab065-swap vg01 -wi-a- 8.00g
  ls8olc04-disk vg01 -wi-a- 16.00g
  ls8olc04-swap vg01 -wi-a- 16.00g
  lvswift vgswift -wi-a- 200.00g
ls8panic: leaved.
Begin: Running /scripts/local-premount ... done.
[ 78.273712] EXT3-fs (dm-20): recovery required on readonly filesystem
[ 78.275821] EXT3-fs (dm-20): write access will be enabled during recovery
[ 78.600757] kjournald starting. Commit interval 5 seconds
[ 78.600805] EXT3-fs (dm-20): recovery complete
[ 78.601250] EXT3-fs (dm-20): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
fsck from util-linux 2.20.1

[Dimitri John Ledkov (xnox)]

While I do have nested LVMs myself, I never actually boot off nested LVM. E.g. the inner vg is presented as top-level for the VM.
Sounds like a very interesting bug.

[Andreas Greve (andreas-greve)]

I don't now much about udev works. But it starts with the highest partition (sdb7 vg: vgswift lv: lvswift as dm-0) of the second disk which is configured for lvm and takes than the next lower partition (sdb6 vg: nova-volumes lv: nova-disk dm-1) and so on. Ok one reason for the working order could be that the first disk sda is an elder SATA-I ST32500N and the disk sdb is an newer SATA-III WD1003FBYX-0.

The "critical" inner vg/lv was created by the following commands.

lvcreate -L 30G -n nova-disk nova-volumes

SR="$( xe sr-create type=ext device-config:device=/dev/nova-volumes/nova-disk name-label=ext )"
POOL="$( xe pool-list --minimal )"
xe pool-param-set uuid=${POOL} default-SR=${SR}

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martijn Lievaart (j-launchpad-net-rtij-nl)]

My problem seems exactly the same, except as far as I can see all symlinks are created. Interestingly, a vgscan -y -a at the command prompt returns immediately, without error.

I do have a rather elaborate setup, with lot's of LVs containing either filesystems or PVs (yes, PVs in LVs) and some snapshots of LVs.