fcp multipath devices does not exist after reboot

[bugproxy]

[Impact]

 * Both systemd units and init scripts are shipped for multipath-tools.

 * However units did not correctly take priority, as initscripts and units names did not match.

 * Further more multipath-tools was miscompiled without systemd support for Type=Notify unit

[Test Case]

 * Using service command, init script, or systemctl commands starting "multipath-tools" should operate on systemd "multipathd.service" unit.

 * Systems must boot with multipath-tools installed correctly, and multipathd.service started.

== Comment: #0 - Ofer Baruch <email address hidden> - 2016-06-06 06:37:19 ==
Hi,

i have installed an Ubuntu Server from the installation ISO.
i then installed multipath support (multipath-tools and multipath-tools-boot).
i added an FCP device, format it, mount it and added to the fstab using it's /dev/mapper representation. everything worked fine...
i then reboot the server and it failed to start due to the missing /dev/mapper device that needs to be mounted (as the fstab states).
i get into maintenance mode.

i am not using cio_ignore and the devices are available in maintenance mode.
running multipath -ll will fail to show the device (as expected... that is why the boot failed).
running multipat without any parameters (to rescan) will find the devices and it will now show on multipath -ll.

i was able to get around this by rebuilding the initrd and running zipl... but that should not be the case. multipath should scan during reboot to find all the devices available.

i am attaching both an sosreport and dbginfo for you to review.
i hope this is something i am doing wrong but i don't think so...
either way i would love to get a quick workaround that will avoid this issue without rebuilding the initrd.

== Comment: #4 - Hendrik Brueckner <email address hidden> - 2016-06-06 06:44:53 ==
Hi Ofer,

(In reply to comment #0)
> i have installed an Ubuntu Server from the installation ISO.
> i then installed multipath support (multipath-tools and
> multipath-tools-boot).
> i added an FCP device, format it, mount it and added to the fstab using it's

How did you add the FCP device?

== Comment: #5 - Ofer Baruch <email address hidden> - 2016-06-06 07:07:42 ==
using:

chzdev zfcp-host <fcp_vdev> -e -p

that is after i vary the device online.
as i said the device comes online just fine... i can see all the /dev/sd* devices.
i am just missing the multipath device.

== Comment: #6 - Ofer Baruch <email address hidden> - 2016-06-06 07:11:09 ==
the missing file is:
/dev/mapper/3600507630bffc0470000000000002333

[bugproxy (bugproxy)]

Default Comment by Bridge

[Dimitri John Ledkov (xnox)]

Are you trying to (a) install ubuntu onto multipath device as root file system or (b) post-install, add and activate a multipath zhcp device for arbitrary data storage?

Reading the installer logs, I see following observations.

The installation was not performed onto a multipath device.

It is recommended to use the installer, and install onto multipath device directly, if that is desired. The installation was performed onto a ECKD dasd drive 0192.

Typically, one is presented with the zhcp screen to activate zhcp devices, and then choose an assembled multipath device, at install time. Then it is correctly setup, in fstab, in the initramfs, in zipl.conf and the generated bootmap with zipl.

See following:
May 11 05:54:40 kernel: [ 153.539344] dasd-eckd 0.0.0192: A channel path to the device has become operational
May 11 05:54:40 kernel: [ 153.541193] dasd-eckd 0.0.0192: New DASD 3390/0C (CU 3990/01) with 2885 cylinders, 15 heads, 224 sectors
May 11 05:54:40 kernel: [ 153.554997] dasd-eckd 0.0.0192: DASD with 4 KB/block, 2077200 KB total size, 48 KB/track, compatible disk layout
May 11 05:54:40 kernel: [ 153.561502] dasda:VOL1/ 0X0192: dasda1 dasda2 dasda3
May 11 05:54:43 s390-dasd[5459]: Printing hashmark every 10 cylinders.
May 11 05:54:43 s390-dasd[5459]:
May 11 05:55:08 kernel: [ 180.968672] dasd-eckd 0.0.0192: DASD with 4 KB/block, 2077200 KB total size, 48 KB/track, compatible disk layout
May 11 05:55:08 kernel: [ 180.968678] dasda: detected capacity change from 0 to 2127052800
May 11 05:55:09 s390-dasd[5459]:
May 11 05:55:09 s390-dasd[5459]:
May 11 05:55:09 s390-dasd[5459]: Finished formatting the device.
May 11 05:55:09 s390-dasd[5459]: Rereading the partition table... ok
May 11 05:55:09 kernel: [ 180.969253] dasda:
May 11 05:55:09 kernel: [ 182.258534] dasda:VOL1/ LX0192:
May 11 05:55:30 main-menu[361]: INFO: Menu item 's390-zfcp' selected
May 11 05:55:30 net/hw-detect.hotplug: Detected hotpluggable network interface enc900
May 11 05:55:30 net/hw-detect.hotplug: Detected hotpluggable network interface lo
May 11 05:55:30 s390-zfcp[5490]: DEBUG: DETECT: The zfcp device driver is not loaded
May 11 05:55:30 s390-zfcp[5490]: WARNING **: Could not open directory: /sys/bus/ccw/drivers/zfcp/: No such file or directory
May 11 05:55:31 main-menu[361]: INFO: Menu item 'disk-detect' selected
May 11 05:55:31 anna-install: Installing mmc-modules
May 11 05:55:31 disk-detect: insmod /lib/modules/4.4.0-21-generic/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko
May 11 05:55:31 kernel: [ 203.950555] rdac: device handler registered
May 11 05:55:31 kernel: [ 203.953114] alua: device handler registered
May 11 05:55:31 disk-detect: insmod /lib/modules/4.4.0-21-generic/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko
May 11 05:55:31 disk-detect: insmod /lib/modules/4.4.0-21-generic/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko
May 11 05:55:31 disk-detect: insmod /lib/modules/4.4.0-21-generic/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko
May 11 05:55:31 kernel: [ 203.955593] hp_sw: device handler registered
May 11 05:55:31 kernel: [ 203.958012] emc: device handler registered

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-06-07 01:37 EDT-------
hi,

option b.
the installation was made on dasd...
post-install: add and activate a multipath zfcp device for arbitrary data storage.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0+git1.656f8865-5ubuntu3

---------------
multipath-tools (0.5.0+git1.656f8865-5ubuntu3) yakkety; urgency=medium

  * Cherrypick from debian:
   - Call clean target before rebuilding with systemd support
   - Call systemd in debian/rules. Also adapt init service
     aliasing.
   - Should resolve LP: #1578638, LP: #1589526, LP: #1583563.

 -- Dimitri John Ledkov <email address hidden> Thu, 16 Jun 2016 11:53:33 +0300

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-06-21 01:03 EDT-------
fix was verified and is working properly...
I am now able to add multipath fcp devices without rebuilding the initrd to survive a reboot.

thank you guys.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-06-21 03:10 EDT-------
Canonical, fix is verified with the Yaketty multipath-tools. Does that mean this will also be SRUed on Xenial?

[Brian Murray (brian-murray)]

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.5.0+git1.656f8865-5ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-06-30 01:07 EDT-------
hi everyone,

we have also tested the multipath-tools/0.5.0+git1.656f8865-5ubuntu2.1 and it works fine.

thanks you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-06-30 10:45 EDT-------
(In reply to comment #17)
> hi everyone,
>
> we have also tested the multipath-tools/0.5.0+git1.656f8865-5ubuntu2.1 and
> it works fine.
>
> thanks you!

Confirmed. Works for us, too. Please promote multipath-tools/0.5.0+git1.656f8865-5ubuntu2.1 to xenial.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0+git1.656f8865-5ubuntu2.1

---------------
multipath-tools (0.5.0+git1.656f8865-5ubuntu2.1) xenial; urgency=medium

  * Cherrypick from debian:
   - Call clean target before rebuilding with systemd support
   - Call systemd in debian/rules. Also adapt init service
     aliasing.
   - Resolves LP: #1578638, LP: #1589526, LP: #1583563. Separate bugs, all
    due to same root cause.

 -- Dimitri John Ledkov <email address hidden> Thu, 16 Jun 2016 11:53:33 +0300

[Martin Pitt (pitti)]

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-07-01 08:19 EDT-------
(In reply to comment #21)
> Made new xenial s390x installation and upgraded only the multipath-tools
> package from the standard repositories. Works fine.
> Thanks a lot!

Closed due to above comment.