Comment 7 for bug 1940687

Revision history for this message
Frank Heimes (fheimes) wrote : Re: autoinstall with multipath/LVM disks sometimes leaves the system in a non-bootable stage

After some more investigations I think this is a more severe problem than I initially thought - and btw. not limited to autoinstall, but affects all (FCP/)multipath installations - it just happened after a reboot of a z/VM guest that uses (FCP/)multipath disk storage, too.

I wanted to add some kernel parameters to enable systemd debugging and direct logging to the console (systemd.log_level=debug systemd.log_target=console),
but I wasn't able because I couldn't re-write the bootloader, due to the fact that it couldn't find any kernel or initrd, because /boot is empty (not properly mounted).
(Right now I even do not understand how the system could come up that way...)
And this is not only after an autoinstall with (FCP/)multipath, but also after a normal interactive installation (that I thought were fine, which is not the case due to the empty /boot).

The following output shows some system states if the system came uo and ran into BusyBox (separated by '---') compared to a situation where it pretended to come up /properly/:

ls -l /dev/dm*
brw------- 1 253, 2 /dev/dm-2
brw------- 1 253, 1 /dev/dm-1
brw------- 1 253, 0 /dev/dm-0
---
$ ls -l /dev/dm*
brw-rw---- 1 root disk 253, 0 Aug 23 10:40 /dev/dm-0

dmsetup ls --tree
mpatha-part2 (253:2)
`-mpatha (253:0)
|- (8:16)
|- (8:0)
|- (8:48)
`- (8:32)
mpatha-part1 (253:1)
`-mpatha (253:0)
|- (8:16)
|- (8:0)
|- (8:48)
`- (8:32)
---
$ sudo dmsetup ls --tree
ubuntu--vg-ubuntu--lv (253:0)
 └─ (8:2)

dmsetup ls
mpatha (253:0)
mpatha-part2 (253:2)
mpatha-part1 (253:1)
---
$ sudo dmsetup ls
ubuntu--vg-ubuntu--lv (253:0)

dmsetup status
mpatha: 0 134217728 multipath 2 0 1 0 1 1 A 0 4 2 8:32 A 0 0 1 8:48 A 0 0 1 8:0
A 0 0 1 8:16 A 0 0 1
mpatha-part2: 0 132116480 linear
mpatha-part1: 0 2097152 linear
---
$ sudo dmsetup status
ubuntu--vg-ubuntu--lv: 0 66060288 linear

dmsetup info
Name: mpatha
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 2
Event number: 0
Major, minor: 253, 0
Number of targets: 1
UUID: mpath-36005076306ffd6b60000000000002602
Name: mpatha-part2
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 2
Number of targets: 1
UUID: part2-mpath-36005076306ffd6b60000000000002602
Name: mpatha-part1
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 1
Number of targets: 1
UUID: part1-mpath-36005076306ffd6b60000000000002602
---
$ sudo dmsetup info
Name: ubuntu--vg-ubuntu--lv
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 0
Number of targets: 1
UUID: LVM-Vnda6LhYVr8Yj4OjePUOhF2fv6lbva7Xg2e2v30TJ28M1mwq8nW0i5DewX14B1cs

ls -l /dev/disk/by-id/*
lrwxrwxrwx 1 10 /dev/disk/by-id/wwn-0x6005076306ffd6b60000000000002602
-part2 -> ../../dm-2
lrwxrwxrwx 1 10 /dev/disk/by-id/wwn-0x6005076306ffd6b60000000000002602
-part1 -> ../../dm-1
lrwxrwxrwx 1 10 /dev/disk/by-id/wwn-0x6005076306ffd6b60000000000002602
-> ../../dm-0
lrwxrwxrwx 1 10 /dev/disk/by-id/scsi-SIBM_2107900_75DXP712602-part2 ->
../../sda2
lrwxrwxrwx 1 10 /dev/disk/by-id/scsi-SIBM_2107900_75DXP712602-part1 ->
../../sda1
lrwxrwxrwx 1 9 /dev/disk/by-id/scsi-SIBM_2107900_75DXP712602 -> ../..
/sdd
lrwxrwxrwx 1 10 /dev/disk/by-id/scsi-36005076306ffd6b60000000000002602
-part2 -> ../../dm-2
lrwxrwxrwx 1 10 /dev/disk/by-id/scsi-36005076306ffd6b60000000000002602
-part1 -> ../../dm-1
lrwxrwxrwx 1 10 /dev/disk/by-id/scsi-36005076306ffd6b60000000000002602
-> ../../dm-0
lrwxrwxrwx 1 10 /dev/disk/by-id/lvm-pv-uuid-Rn4VgD-Nr5e-0zK9-xJ9t-4RNK
-Pena-VE9VqC -> ../../dm-2
lrwxrwxrwx 1 10 /dev/disk/by-id/dm-uuid-part2-mpath-36005076306ffd6b60
000000000002602 -> ../../dm-2
lrwxrwxrwx 1 10 /dev/disk/by-id/dm-uuid-part1-mpath-36005076306ffd6b60
000000000002602 -> ../../dm-1
lrwxrwxrwx 1 10 /dev/disk/by-id/dm-uuid-mpath-36005076306ffd6b60000000
000002602 -> ../../dm-0
lrwxrwxrwx 1 10 /dev/disk/by-id/dm-name-mpatha-part2 -> ../../dm-2
lrwxrwxrwx 1 10 /dev/disk/by-id/dm-name-mpatha-part1 -> ../../dm-1
---
$ ls -l /dev/disk/by-id/*
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/dm-name-ubuntu--vg-ubuntu--lv -> ../../dm-0
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/dm-uuid-LVM-Vnda6LhYVr8Yj4OjePUOhF2fv6lbva7Xg2e2v30TJ28M1mwq8nW0i5DewX14B1cs -> ../../dm-0
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/lvm-pv-uuid-Rn4VgD-Nr5e-0zK9-xJ9t-4RNK-Pena-VE9VqC -> ../../sdd2
lrwxrwxrwx 1 root root 9 Aug 23 09:28 /dev/disk/by-id/scsi-36005076306ffd6b60000000000002602 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/scsi-36005076306ffd6b60000000000002602-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/scsi-36005076306ffd6b60000000000002602-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 9 Aug 23 09:28 /dev/disk/by-id/scsi-SIBM_2107900_75DXP712602 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/scsi-SIBM_2107900_75DXP712602-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/scsi-SIBM_2107900_75DXP712602-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 9 Aug 23 09:28 /dev/disk/by-id/wwn-0x6005076306ffd6b60000000000002602 -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/wwn-0x6005076306ffd6b60000000000002602-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Aug 23 09:28 /dev/disk/by-id/wwn-0x6005076306ffd6b60000000000002602-part2 -> ../../sdd2

ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 10 2f3ae014-30be-4d0f-a8f6-e4d3ad23baa9 -> ../../dm-1
---
$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Aug 23 10:40 2f3ae014-30be-4d0f-a8f6-e4d3ad23baa9 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Aug 23 10:40 89e16fd3-e092-4ab2-b15d-5fede7c92de7 -> ../../dm-0

ls -l /dev/mapper
crw------- 1 10, 236 control
lrwxrwxrwx 1 7 mpatha -> ../dm-0
lrwxrwxrwx 1 7 mpatha-part2 -> ../dm-2
lrwxrwxrwx 1 7 mpatha-part1 -> ../dm-1
---
$ ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 Aug 23 10:40 control
lrwxrwxrwx 1 root root 7 Aug 23 10:40 ubuntu--vg-ubuntu--lv -> ../dm-0

That seems to be broken in both cases ...