lvm2-pvscan services fails to start on S390X DPM

[Lee Trager]

The lvm2-pvscan service fails to start in the MAAS ephemeral environment. The service takes ~3 minutes to fail, blocking boot until failure. Curtin is expereincing a similar bug(LP:1813228) due to LVM.

Release: 18.10(18.04 does not currently boot on S390X DPM)
Kernel: 4.18.0-13-generic
LVM: 2.02.176-4.1ubuntu3

root@node3:~# lvm pvscan --cache --activate ay 8:98
  WARNING: Not using lvmetad because duplicate PVs were found.
  WARNING: Autoactivation reading from disk instead of lvmetad.
  WARNING: PV tIvQUe-wsuE-RWlZ-Wmdo-uzws-qL5H-Y9zW5l on /dev/sdb2 was already found on /dev/sdfe2.
  WARNING: PV 3zsO6H-QCQq-eyJZ-8aLc-TGU1-q91J-LFWsKs on /dev/sde2 was already found on /dev/sdfh2.
  WARNING: PV VEyfqe-Vln5-zMp5-wCeK-hkQN-u8Wp-LkPTMF on /dev/sdba2 was already found on /dev/sdhd2.
  WARNING: PV H7Ucbo-TJSw-tS4y-AwKT-xdql-NiMp-YvZC1S on /dev/sdfj2 was already found on /dev/sddi2.
  WARNING: PV H7Ucbo-TJSw-tS4y-AwKT-xdql-NiMp-YvZC1S on /dev/sdg2 was already found on /dev/sddi2.
  WARNING: PV tIvQUe-wsuE-RWlZ-Wmdo-uzws-qL5H-Y9zW5l on /dev/sdbc2 was already found on /dev/sdfe2.
  WARNING: PV 3zsO6H-QCQq-eyJZ-8aLc-TGU1-q91J-LFWsKs on /dev/sdbf2 was already found on /dev/sdfh2.
  WARNING: PV VEyfqe-Vln5-zMp5-wCeK-hkQN-u8Wp-LkPTMF on /dev/sddb2 was already found on /dev/sdhd2.
  WARNING: PV 6ZwSAD-cFkP-Qjeg-WIDt-g3Hc-m8Tt-i3ex81 on /dev/sddr2 was already found on /dev/sdbq2.
  WARNING: PV H7Ucbo-TJSw-tS4y-AwKT-xdql-NiMp-YvZC1S on /dev/sdbh2 was already found on /dev/sddi2.
  WARNING: PV tIvQUe-wsuE-RWlZ-Wmdo-uzws-qL5H-Y9zW5l on /dev/sddd2 was already found on /dev/sdfe2.
  WARNING: PV 3zsO6H-QCQq-eyJZ-8aLc-TGU1-q91J-LFWsKs on /dev/sddg2 was already found on /dev/sdfh2.
  WARNING: PV VEyfqe-Vln5-zMp5-wCeK-hkQN-u8Wp-LkPTMF on /dev/sdfc2 was already found on /dev/sdhd2.
  WARNING: PV 6ZwSAD-cFkP-Qjeg-WIDt-g3Hc-m8Tt-i3ex81 on /dev/sdfs2 was already found on /dev/sdbq2.
  WARNING: PV 6ZwSAD-cFkP-Qjeg-WIDt-g3Hc-m8Tt-i3ex81 on /dev/sdp2 was already found on /dev/sdbq2.
  WARNING: PV tIvQUe-wsuE-RWlZ-Wmdo-uzws-qL5H-Y9zW5l prefers device /dev/sdfe2 because device was seen first.
  WARNING: PV tIvQUe-wsuE-RWlZ-Wmdo-uzws-qL5H-Y9zW5l prefers device /dev/sdfe2 because device was seen first.
  WARNING: PV tIvQUe-wsuE-RWlZ-Wmdo-uzws-qL5H-Y9zW5l prefers device /dev/sdfe2 because device was seen first.
  WARNING: PV 3zsO6H-QCQq-eyJZ-8aLc-TGU1-q91J-LFWsKs prefers device /dev/sdfh2 because device was seen first.
  WARNING: PV 3zsO6H-QCQq-eyJZ-8aLc-TGU1-q91J-LFWsKs prefers device /dev/sdfh2 because device was seen first.
  WARNING: PV 3zsO6H-QCQq-eyJZ-8aLc-TGU1-q91J-LFWsKs prefers device /dev/sdfh2 because device was seen first.
  WARNING: PV VEyfqe-Vln5-zMp5-wCeK-hkQN-u8Wp-LkPTMF prefers device /dev/sdhd2 because device was seen first.
  WARNING: PV VEyfqe-Vln5-zMp5-wCeK-hkQN-u8Wp-LkPTMF prefers device /dev/sdhd2 because device was seen first.
  WARNING: PV VEyfqe-Vln5-zMp5-wCeK-hkQN-u8Wp-LkPTMF prefers device /dev/sdhd2 because device was seen first.
  WARNING: PV H7Ucbo-TJSw-tS4y-AwKT-xdql-NiMp-YvZC1S prefers device /dev/sddi2 because device was seen first.
  WARNING: PV H7Ucbo-TJSw-tS4y-AwKT-xdql-NiMp-YvZC1S prefers device /dev/sddi2 because device was seen first.
  WARNING: PV H7Ucbo-TJSw-tS4y-AwKT-xdql-NiMp-YvZC1S prefers device /dev/sddi2 because device was seen first.
  WARNING: PV 6ZwSAD-cFkP-Qjeg-WIDt-g3Hc-m8Tt-i3ex81 prefers device /dev/sdbq2 because device was seen first.
  WARNING: PV 6ZwSAD-cFkP-Qjeg-WIDt-g3Hc-m8Tt-i3ex81 prefers device /dev/sdbq2 because device was seen first.
  WARNING: PV 6ZwSAD-cFkP-Qjeg-WIDt-g3Hc-m8Tt-i3ex81 prefers device /dev/sdbq2 because device was seen first.
  Cannot activate LVs in VG zkvm4 while PVs appear on duplicate devices.
  0 logical volume(s) in volume group "zkvm4" now active
  zkvm4: autoactivation failed.
  Cannot activate LVs in VG zkvm1 while PVs appear on duplicate devices.
  0 logical volume(s) in volume group "zkvm1" now active
  zkvm1: autoactivation failed.
  Cannot activate LVs in VG Z_APPL_ROOT_lvm_473D86E3 while PVs appear on duplicate devices.
  0 logical volume(s) in volume group "Z_APPL_ROOT_lvm_473D86E3" now active
  Z_APPL_ROOT_lvm_473D86E3: autoactivation failed.
  Cannot activate LVs in VG Z_APPL_ROOT_lvm_BBD0E8D4 while PVs appear on duplicate devices.
  0 logical volume(s) in volume group "Z_APPL_ROOT_lvm_BBD0E8D4" now active
  Z_APPL_ROOT_lvm_BBD0E8D4: autoactivation failed.
  Cannot activate LVs in VG Z_APPL_ROOT_lvm_382F5B8E while PVs appear on duplicate devices.
  0 logical volume(s) in volume group "Z_APPL_ROOT_lvm_382F5B8E" now active
  Z_APPL_ROOT_lvm_382F5B8E: autoactivation failed.
root@node3:~# echo $?
5

[Christian Ehrhardt  (paelzer)]

Hi,
I'm not sure what exactly will be needed, but the following would surely help.
Do you have a way to fetch the following while at the error condition and attach it here?:
- sudo multipath -ll
- lsblk

[Lee Trager (ltrager)]

The MAAS ephemeral environment does not have the multipath kernel module nor multipath-tools. To get this log I had to install both.

[Christian Ehrhardt  (paelzer)]

Hi Lee,
thanks for the logs - once multipath is installed the devices are multipath mapped.
But obviously at the time pvscan failed they were not.

I'd expect that after you have installed all dependencies for multipath to work that then restarting lvm2-pvscan works fine.

Can you confirm that this is the case?
If it is then the solution is either:
- have multipath avaiable in your ephemeral environment if you need it's support
- make the scan ignore some devices

I'd need a deeper look but those seem as the most likely reasons

[Christian Ehrhardt  (paelzer)]

I saw [1] by xnox popping up, so he might work on this already. I have pinged him to clarify.

[1]: https://code.launchpad.net/~xnox/ubuntu-seeds/+git/ubuntu-seeds/+merge/363541

[Christian Ehrhardt  (paelzer)]

If the change by xnox fixes your case then the resolution is to ship multipath on those images.
@Lee - waiting for your answer if the services (and thereby the pvscan) works after multipath is installed.

In general I think xnox might work on this then (not sure)?
I subscribed him to be aware as this is not tagged up as s390x related yet.

[Christian Ehrhardt  (paelzer)]

I also pinged jfh to make it part of the s390x related tracking

[Lee Trager (ltrager)]

I tested a Disco image today with multipath-tools installed and I still get the same thing.