Curtin is creating inconsistent name for mpath device for ppc64el

[Tiago Mello]

For Power8 deployments, MaaS is creating inconsistent name for mpath... while setup uses letters, the petitboot entry is using numbers, like: mpath0 should mpatha instead.

[Andres Rodriguez (andreserl)]

Hi Tiago,

MAAS does not set nor configures multipath. This iss all done by curtin.

Chaging this as such!

[Ryan Harper (raharper)]

Please attach the curtin configuration maas sends to the machine.

maas <session> node get-curtin-config <system-id>

[Ryan Harper (raharper)]

Could you also provide your petitboot entries that are incorrect and indicate what they should be?

[Scott Moser (smoser)]

Can you please explain why this inconsistency is a problem? Petitboot needs to find the kernel, imitramfs and command line and kexec. I don't understand why it would have to care what the voted kernel calls the multipath device.

[Rafael Folco (rafaelfolco)]

I see 2 problems with multipath and LVM:

1) friendly_names are mpath[a-z], not mpath[0-9]
2) hardware raid arrays (iprconfig) named as 1IBM_IPR-0_XXXXXXXX

This controller has dual SAS ports, which duplicates disk paths and enables multipath. This setup uses RAID0 for the IPR disks, which gives us IO performance enhancements.

I recreated the RAID array with iprconfig, adding all disks to the RAID0 array. Removed the node from MaaS (since re-comissioning did not refresh disk settings correctly), and re-added to MaaS. Restarted comissioning and it recognizes now the array as a single disk with 3TB. Deployed Xenial, and got stuck on initramfs:

<system boot after deploy, it drops to busybox>

Begin: Running /scripts/local-block ... lvmetad is not active yet, using direct activation during sysinit
  Volume group "mpath0" not found
  Cannot process volume group mpath0
done.
Begin: Running /scripts/local-block ... lvmetad is not active yet, using direct activation during sysinit
  Volume group "mpath0" not found
  Cannot process volume group mpath0
done.
done.
Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/mpath0-part2 does not exist. Dropping to a shell!
[ 258.070061] hidraw: raw HID events driver (C) Jiri Kosina
[ 258.071998] usbcore: registered new interface driver usbhid
[ 258.072047] usbhid: USB HID core driver

BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

(initramfs) cat /proc/cmdline
root=/dev/mapper/mpath0-part2 ro disk-detect/multipath/enable=true net.ifnames=1 biosdevname=0

(initramfs) ls /dev/mapper
control 1IBM_IPR-0_5EC63900000001E0-part1
1IBM_IPR-0_5EC63900000001E0-part2

(initramfs) cat /etc/multipath.conf
# This file was created by curtin while installing the system.
defaults {
 user_friendly_names yes
}

(initramfs) cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/1IBM_IPR-0_5EC63900000001E0/

(initramfs) /sbin/blkid
/dev/sda2: LABEL="root" UUID="a4fa6afa-c553-4cfc-ae6f-82e0637edd5a" TYPE="ext4" PARTUUID="723d18ee-be3b-4e28-84fa-2d444d69c174"
/dev/sdb2: LABEL="root" UUID="a4fa6afa-c553-4cfc-ae6f-82e0637edd5a" TYPE="ext4" PARTUUID="723d18ee-be3b-4e28-84fa-2d444d69c174"
/dev/mapper/1IBM_IPR-0_5EC63900000001E0-part2: LABEL="root" UUID="a4fa6afa-c553-4cfc-ae6f-82e0637edd5a" TYPE="ext4" PARTUUID="723d18ee-be3b-4e28-84fa-2d444d69c174"
/dev/sda1: PARTUUID="f5089e1b-cd09-4018-a0b4-6058474c2caa"
/dev/sdb1: PARTUUID="f5089e1b-cd09-4018-a0b4-6058474c2caa"
/dev/dm-0: PTUUID="5df0b342-bbc9-478d-b75a-e77308ceaa26" PTTYPE="gpt"
/dev/mapper/1IBM_IPR-0_5EC63900000001E0-part1: PARTUUID="f5089e1b-cd09-4018-a0b4-6058474c2caa"

(initramfs) /sbin/multipath -ll
1IBM_IPR-0_5EC63900000001E0 dm-0 IBM,IPR-0 5EC63900
size=4.2T features='1 queue_if_no_p...

[Tiago Mello (timello)]

Also,

Looks like mpath0 is hardcoded in curtin? Please, see:

http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/curtin/commands/curthooks.py#L624

[Tiago Mello (timello)]

This patch fixed the problem.

[Ryan Harper (raharper)]

Tiago,

Thanks for the patch.

I'd prefer to extract the name from multipathd or device-mapper itself, rather than assume we know how either will treat names with spaces. Different versions of multipath on other releases may have different behavior on how they shorten the name or deal with spaces.

In curtin, we're extracting a WWID from the block device itself, which has a WWID with spaces. However, multipath itself should have written out /etc/multipath/wwids and/or /etc/multipath/bindings; or at least it appears that one is written into your initrd.

Can you attach those files to the bug?

[Tiago Mello (timello)]

Those are the files after the deployment (with the fix). I'll re-deploy without the patch above and attach what initrd has within it.

# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/1IBM_IPR-0_5ED5AD0000000060/

# This file was created by curtin while installing the system.
mpath0 1IBM_IPR-0_5ED5AD0000000060
# End of content generated by curtin.
# Everything below is maintained by multipath subsystem.

[Tiago Mello (timello)]

(initramfs) cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/1IBM_IPR-0_5ED5AD0000000060/

(initramfs) cat /etc/multipath/bindings
# This file was created by curtin while installing the system.
mpath0 1IBM IPR-0 5ED5AD0000000060
# End of content generated by curtin.
# Everything below is maintained by multipath subsystem.