multipathd drops paths of a temporarily lost device

[bugproxy]

== Comment: #0 - Thorsten Diehl <email address hidden> - 2016-02-01 08:57:28 ==
# uname -a
Linux s83lp31 4.4.0-1-generic #15-Ubuntu SMP Thu Jan 21 22:19:04 UTC 2016 s390x s390x s390x GNU/Linux

# dpkg -s multipath-tools|grep ^Version:
Version: 0.5.0-7ubuntu9

# cat /etc/multipath.conf
defaults {
    default_features "1 queue_if_no_path"
    user_friendly_names yes
    path_grouping_policy multibus
    dev_loss_tmo 2147483647
    fast_io_fail_tmo 5
}

blacklist {
    devnode '*'
}

blacklist_exceptions {
    devnode "^sd[a-z]+"
}

---------------------------------------
On a z Systems LPAR with a single LUN, 2 zfcp devices, 2 storage ports, and the following multipath topology:

mpatha (36005076304ffc3e80000000000003050) dm-0 IBM,2107900
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 0:0:0:1079001136 sda 8:0 active ready running
  |- 0:0:1:1079001136 sdb 8:16 active ready running
  |- 1:0:0:1079001136 sdc 8:32 active ready running
  `- 1:0:1:1079001136 sdd 8:48 active ready running

I observed the following:
When I deconfigure one of the two zfcp devices (e.g. via chchp -c 0, or directly on the HMC), the multipathd removes the two paths via these devices from the pathgroup after 10 seconds. When the zfcp devices comes back, it runs through zfcp error recovery and is being set up properly, and also the mid layer objects are looking fine. However, the multipathd does not add them to the path group again.

Expected behaviour: multipathd does not remove the paths from topology list, but holds them as "failed faulty offline" until dev_loss_tmo timout is reached (which is infinite here).

I discussed this already with zfcp development, and it looks most likely as a problem with multipathd, rather than zfcp or mid-layer.

Easy to reproduce: you need two zfcp devices, one LUN, and preferably two ports on the storage server (WWPNs). Configure LUN via 2 zfcp devices * 2 WWPNs = 4 paths.

This can be also reproduced on a z/VM guest. Instead of configuing the CHPID off, just detach one zfcp device and re-attach it after 30....60 seconds. Same problem.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1540407/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Thorsten Diehl (thorsten-diehl)]

Hi Canonical Server Team, on another distro with a previous version of multipath-tools (0.4.9) I can not see this misbehaviour.
Although 0.5.0 isn't that new, I don't know of another distro for s390x with 0.5.0 where I could check this.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-24 09:36 EDT-------
Tested on a debian system:
Linux s8330004 4.3.0-1-s390x #1 SMP Debian 4.3.5-1 (2016-02-06) s390x GNU/Linux (stretch) +
multipath-tools 0.5.0+git1.656f8865-4
==> problem does not occur!!

Linux s8330004 4.4.0-1-s390x #1 SMP Debian 4.4.2-3 (2016-02-21) s390x GNU/Linux (sid) +
multipath-tools 0.5.0+git1.656f8865-4 (unchanged)
==> problem does not occur!!

multipath.conf see above
will attach outputs of multipath -t of ubuntu and debian system for comparison.

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2016-02-24 09:37 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2016-02-24 09:38 EDT-------

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-24 11:11 EDT-------
Problem persists with Ubuntu kernel 4.4.0-7.

[Ryan Harper (raharper)]

Hi Thorsten,

The latest version of the multipath-tools package is 0.5.0-7ubuntu14. Can you confirm you're still seeing the issue?

Meanwhile, I'm hoping to recreate this issue on zKVM shortly. In the meantime I'm testing this on in an x86 VM with multipath via virtio-scsi, using the same multipath.conf as mentioned in the bug.

# dpkg -s multipath-tools| grep ^Version
Version: 0.5.0-7ubuntu14

# multipath -ll
mpatha (0QEMU_QEMU_HARDDISK_0001) dm-0 QEMU,QEMU HARDDISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 2:0:0:0 sda 8:0 active ready running
  `- 2:0:0:1 sdb 8:16 active ready running

I can mark the device offline with:

# echo "offline" > /sys/class/block/sdb/device/state
# multipath -ll
mpatha (0QEMU_QEMU_HARDDISK_0001) dm-0 QEMU,QEMU HARDDISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 2:0:0:0 sda 8:0 active ready running
  `- 2:0:0:1 sdb 8:16 active faulty offline

# sleep 60 && multipath -ll
mpatha (0QEMU_QEMU_HARDDISK_0001) dm-0 QEMU,QEMU HARDDISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 2:0:0:0 sda 8:0 active ready running
  `- 2:0:0:1 sdb 8:16 failed faulty offline

And bring it back online with:

# echo "1" > /sys/block/sdb/device/delete
# for RESCAN in /sys/class/scsi_host/*; do echo "- - -" > $RESCAN/scan; done
# multipath -v2
# multipath -ll
mpatha (0QEMU_QEMU_HARDDISK_0001) dm-0 QEMU,QEMU HARDDISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 2:0:0:0 sda 8:0 active ready running
  `- 2:0:0:1 sdb 8:16 active ready running

It maybe that the underlying device would need to be in error instead of the scsi layer in the kernel.
I'll update this when I get the zKVM instance up with multipath as described in the bug.

Looking at the delta between 0.5.0 in Ubuntu and the newer version in Debian, there are a number of changes in the area of discovery and path checking which may resolve this issue if we can confirm we're still seeing the issue on the latest version in Xenial.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-25 06:05 EDT-------
Hi Ryan,

on the most recent repro I already had version: 0.5.0-7ubuntu14 - same symptom.
With
# echo "offline" > /sys/class/block/sdb/device/state
I could not reproduce this symptom on z Systems.

Approved repro is as follows:
z Systems LPAR: chchp -c 0 <CHPID>;sleep 30;chchp -c 1 <CHPID>
z Systems z/VM guest: vmcp det <fcpdevice>;sleep 30;vmcp att <fcpdevice> "*"

[Ryan Harper (raharper)]

Hi Thorsten,

Thanks for the update. I'll attempt to find out if I can force a detach on the host-side to recreate the issue on x86 while I'm waiting for a Z system with FCP devices.

Could you attempt to use the Debian package from Sid on top of the Ubuntu system? This will help rule out kernel related issues vs. userspace package bugs.

You can download from here

http://ftp.us.debian.org/debian/pool/main/m/multipath-tools/multipath-tools_0.5.0+git1.656f8865-4_s390x.deb

And then force install with

sudo dpkg --force all --install multipath-tools_0.5.0+git1.656f8865-4_s390x.deb

If this is successful, then I can prepare a package with some updated fixes from upstream; there are a number of changes to path related discovery in upstream multipath that aren't yet included in the Ubuntu version.

[Christian Ehrhardt  (paelzer)]

Hi,
I couldn't reproduce today.

The device stays in:
0:0:0:1074675728 sdb 8:16 active faulty offline
And when re-adding it comes back online just nice.

The way I configured was via chzdev like:
I used the z/VM approach with detaching the FCP adapter.
I used the multipath.conf as reported initially.
chzdev zfcp --enable 7200
chzdev zfcp --enable 7300
chzdev zfcp-lun --enable --online <- warning I enable ALL, you might want just some of them

We realized as difference to Thorstens systems I also had from this ppa: sudo add-apt-repository ppa:xnox/nonvir
From here I installed packages:
libzfcphbaapi0
zfcp-hbaapi-utils

Versions:
libzfcphbaapi0 2.1.1-0ubuntu1
zfcp-hbaapi-utils 2.1.1-0ubuntu1
multipath-tools-boot 0.5.0-7ubuntu15
Kernel 4.4.0-8-generic

I had a call with Thorsten and he will re-verify with the latest version on his system.
If his case still fails he might give it a try with the "way I configured it".

Other than that Thorsten suggested he would like it to see the merge happening to the more recent upstream content that Ryan mentioned.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-29 06:27 EDT-------
(In reply to comment #16)
> > Could you attempt to use the Debian package from Sid on top of the Ubuntu
> system? This will help rule out kernel related issues vs. userspace package
> bugs.
>
> You can download from here
>
> http://ftp.us.debian.org/debian/pool/main/m/multipath-tools/multipath-
> tools_0.5.0+git1.656f8865-4_s390x.deb
>
> And then force install with
>
> sudo dpkg --force all --install
> multipath-tools_0.5.0+git1.656f8865-4_s390x.deb
>
> If this is successful, then I can prepare a package with some updated fixes
> from upstream; there are a number of changes to path related discovery in
> upstream multipath that aren't yet included in the Ubuntu version.

Hi Ryan,

I followed your proposal, installed multipath-tools_0.5.0+git1.656f8865-4_s390x.deb as described, and it worked fine and as expected, i.e. described problem does not occur.
On another system I installed both multipath-tools_0.5.0+git1.656f8865-4_s390x.deb and kpartx_0.5.0+git1.656f8865-4_s390x.deb - with the same result.
I did a detach/attach of the zfcp device and found, that it apperas for some seconds in status active faulty running, then it changed to failed faulty running and remained in that state until reattachment of the device. Then it changed back to active ready running.

Hi Christian,

in addition I did the following tests:

1. On a freshly installed xenial (installer version 427, kernel 4.4.0-8) with the already mentioned multipath.conf I found, that no zfcphbaapi stuff is installed per default.
kernel 4.4.0-8
multipath-tools 0.5.0-7ubuntu15 (was 7ubuntu14 before) and kpartx 0.5.0-7ubuntu15
I did a detach/attach of the zfcp device and found, that it apperas for some seconds in status active faulty offline, then it changed to failed faulty offline and vanished and did not reappear upon reattachment of the device. I had the zfcp LUNs configure manually (via sysfs)

2. Then I installed zfcp-hbaapi-utils_2.1.1-0ubuntu1_s390x.deb and libzfcphbaapi0_2.1.1-0ubuntu1_s390x.deb on top. With these two additional packages I could not reproduce this problem.

3. I installed a system with kernel 4.4.0-7 and multipath-tools 0.5.0-7ubuntu14 + the above mentioned zfcphbaapi stuff. Problem was reproducable.

4. I updated that system to multipath-tools 0.5.0-7ubuntu15. Problem was still reproducable.

5. I updated that system to kernel 4.4.0-8. It should be now similar to case #2. But problem was still reproducable. strange.

6. I switched several times back and forth between multipath-tools 0.5.0-7ubuntu15 and multipath-tools_0.5.0+git1.656f8865-4_s390x.deb. With git1 it worked always, with 7ubuntu15 never. (although it did on a freshly installed system, see #2)

So, for now we have two ways to solve/circumvent this problem:
a) use an updated multipath-tools package (from debian sid)
b) use an updated multipath-tools/kpartx packages from ubuntu (>= version 0.5.0-7ubuntu15) AND zfcp-hbaapi-utils_2.1.1-0ubuntu1_s390x.deb and libzfcphbaapi0_2.1.1-0ubuntu1_s390x.deb on top. But that didn't help always (see case #5 above)

I found different values for path_selector in multipath-tools versions of ubunt...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-29 08:50 EDT-------
Regarding the above strange behaviour in #5:
On a 4.4.0-8 system freshly installed (not updated from 4.4.0-7) with libzfcphbaapi0 installed the paths are recoverd and re-added as expected, as already described in case #2. And again, with case #1 the problem is reproducable.

What is this libzfcphbaapi0 package doing undercover? Normally it has no interatice with multipath daemon. Does it change anything in the multipath configuration?

Adding the libzfcphbaapi0 package to prevent symptoms of this bug can not be the solution and it might be just by chance, that the symptom vanished. I talked to zfcp development, and they confirmed that.

As already promised, I will continue some long term tests (cable pull and other path loosing scenarios) on a 4.4.0-8 system with multipath-tools_0.5.0+git1.656f8865-4 and kpartx_0.5.0+git1.656f8865-4. Results should be available by tomorrow.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-01 03:58 EDT-------
(In reply to comment #19)
> As already promised, I will continue some long term tests (cable pull and
> other path loosing scenarios) on a 4.4.0-8 system with
> multipath-tools_0.5.0+git1.656f8865-4 and kpartx_0.5.0+git1.656f8865-4.
> Results should be available by tomorrow.

Results show, that this combination does not harm, instead it solves the subject problem, even after 100 cycles.

Ryan, but it's up to you what you are going to integrate to fix this bug.

[Ryan Harper (raharper)]

On Tue, Mar 1, 2016 at 3:00 AM, bugproxy <email address hidden> wrote:

> ------- Comment From <email address hidden> 2016-03-01 03:58 EDT-------
> (In reply to comment #19)
> > As already promised, I will continue some long term tests (cable pull and
> > other path loosing scenarios) on a 4.4.0-8 system with
> > multipath-tools_0.5.0+git1.656f8865-4 and kpartx_0.5.0+git1.656f8865-4.
> > Results should be available by tomorrow.
>
> Results show, that this combination does not harm, instead it solves the
> subject problem, even after 100 cycles.
>
> Ryan, but it's up to you what you are going to integrate to fix this
> bug.
>

Thorsten,

Thanks for the thorough testing of the scenarios. I'm currently examining
the changes
introduced into the newer Debian package and looking at which changes we'd
need
to fix this bug and also understand if a merge of the package makes more
sense.

Ryan

>
> --
> You received this bug notification because you are a member of Canonical
> Server Team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1540407
>
> Title:
> multipathd drops paths of a temporarily lost device
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540407/+subscriptions
>

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-02 02:36 EDT-------
Ryan,

I appreaciate to read that, that's the best approach.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-08 09:40 EDT-------
Ryan, what's your decision regarding this defect? Since I have to repeat many tests with that fixed multipath version, I'm really looking forward to get it soon.

[Christian Ehrhardt  (paelzer)]

FYI - stalled on FCP setup in RT 89162

[Ryan Harper (raharper)]

Hi Thorsten,

We're going to merge with upstream debian. That work is tracked under bug 1551952.

I'm now working on getting an s390x build of the package and reproducing on one of the systems to
help test that the merge addresses this bug (and doesn't regress anything as well).

[Ryan Harper (raharper)]

Thorsten,

Here's my PPA with the merged multipath-tools package for s390x:

https://launchpad.net/~raharper/+archive/ubuntu/merges/

You can add it via:

sudo add-apt-repository ppa:raharper/merges
sudo apt-get update
sudo apt-cache policy multipath-tools

The last command should show you a newer version from the ppa as Candidate.

Then you can update the multpath-tools binaries (and kpartx friends) with:

sudo apt-get install kpartx kpartx-boot multipath-tools multipath-tools-boot

This package in the PPA is not in final form; but any testing/feedback on it will help us
close any issues with the merge to latest debian.

[Ryan Harper (raharper)]

Here's my initial test with the merged multipath-tools test.
I had the FCP devices enabled with the 0.5.0+git<hash>-1ubuntu2 package installed from the merges ppa and rebooted the system.

After booting, I confirmed the paths were up, then used a vmcp command to disconnect the devices.
I then queried mulitpath over a number of minutes to ensure the path remains (but shows faulty).
After 15 minutues or so I contacted an admin to renable the FCP devices and observed the paths
becoming restored in multipath

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-11 05:05 EDT-------
(In reply to comment #27)
> Here's my initial test with the merged multipath-tools test.
> I had the FCP devices enabled with the 0.5.0+git<hash>-1ubuntu2 package
> installed from the merges ppa and rebooted the system.
>
> After booting, I confirmed the paths were up, then used a vmcp command to
> disconnect the devices.
> I then queried mulitpath over a number of minutes to ensure the path remains
> (but shows faulty).
> After 15 minutues or so I contacted an admin to renable the FCP devices and
> observed the paths
> becoming restored in multipath

Ryan,
congrats, well done. :-) I did an extended test with this multipathd/kpartx on a z/VM guest and in LPAR overnight and it ran very well. Even after 300 "off/on" cycles all paths were "active ready running" at the end.
So, please go ahead with this. I will continue testing this when it is in xenial/main, since some of my other tests rely on this.

I will close this defect when this fix is in xenial/main.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-16 06:43 EDT-------
Canonical: This bug is currently in stage "Triaged". Please provide the info , when it will be in /release/main. Many thanks in advance

[Dimitri John Ledkov (xnox)]

The upload with this bugfix has been prepared, however it breaks the Feature Freeze. A feature freeze exception has been requested and is currently awaiting Release Team review. Once/If the release team reviews the feature freeze exception, the package will be uploaded into xenial-proposed. You can track the progress of the Feature Freeze Exception here https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1551952 hence setting the status of this bug report to in progress.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0+git1.656f8865-5ubuntu1

---------------
multipath-tools (0.5.0+git1.656f8865-5ubuntu1) xenial; urgency=medium

  * Merge from Debian. (LP: #1551952) (LP: #1540407)
    Remaining changes:
    - debian/control:
      - Bump debhelper dependency to install udev rules to
        /lib/udev/rules.d
      - Bump udev dependencies as well
    - debian/control:
      - multipath-tools: add sg3-utils-dev Depends
      - multipath-udeb: add sg3-udeb Depends
    - debian/initramfs/hooks: use 95 not 60 for multipath rules priority
    - debian/rules: Move udev rules to priority 95, because rules that load modules should be >90.
    - debian/multipath-tools.preinst: modprobe dm-multipath.
      This will make sure that multipathd will be able to start.
    - debian/initramfs/local-top: wait for udev to settle before running
      'multipath' in order to avoid race condition on device-mapper calls.
    - debian/initramfs/local-top: remove '--timeout 10' which causes my
      test system to not boot roughly 3 out of 4 times.
    - Split kpartx initramfs bits into kpartx-boot for dmraid (LP: #941874)
      - debian/initramfs/kpartx.hook
      - debian/kpartx-boot.postinst
      - debian/kpartx-boot.postrm
      - debian/kpartx.install
      - debian/control: Add kpartx-boot package for dmraid
      - debian/rules: Install kpartx initramfs hook
    - debian/patches/1000--set-umask-in-multipathd.patch: Set umask in multipathd.
    - debian/patches/handle_spaces_in_rev_attr.patch: support IBM IPR devices
      and others which may have only spaces for the rev attribute.
    - debian/patches/path_selector.patch: switch the default path selector
      back to round-robin while service-time isn't available to the installer
      multipath-modules.
    - debian/patches/0015-shared-lock-for-udev.patch: (LP: #1431650)
    - debian/initramfs/hooks: also copy wwids file on the installed system to
      ensure all paths come up on boot. (LP: #1479929)
    - Disable -fexceptions on multipath-udeb (LP: #1489379): the flag causes
      libchecktur.so to link with libgcc_s.so.1 (even with -static-libgcc),
      which is not available in the installer environment.
      - debian/patches/disable-fexceptions-udeb.patch: conditionally disable
        -fexceptions with CFLAGS_DISABLE_FEXCEPTIONS.
      - debian/rules: set CFLAGS_DISABLE_FEXCEPTIONS to build multipath-udeb.
    - debian/patches/handle_spaces_in_rev_attr.patch: update patch to apply the
      change to the right line (LP: #1492425)
    - debian/initramfs/local-premount: wait for udev to settle before the call
      to resolve_device() in local_mount_root(), so the by-uuid/ symlinks have
      a chance to be updated by the multipath udev rules (LP: #1503286).
    - debian/multipath-tools.postinst: handle upgrades from < 0.5.0 by migrating
      from the old device names with device numbers to using letters for devices.
    - debian/patches/mpath_name_migration.patch: ship a multipath_migrate binary
      to make translation from pre-0.5.0 device naming to the new scheme.
    - debian/initramfs/hooks: install multipathd and required directories.
    - debian/i...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-21 13:50 EDT-------
Hi Ryan, hi Dimitri,

Update from LP 1551952:
> St?phane Graber (stgraber) wrote on 2016-03-18: #7
>
> Unfortunate that this only gets in so late in the cycle, but I can certainly
> see the benefit of updating this before we have to support it for 5 years.
>
> Testing seems reasonable. debdiff is unreviewable so we can only rely on the
> testing...
>
> Assuming that you'll take care of any yet unknown regression from this, you
> may go ahead and upload the merge.
> Changed in multipath-tools (Ubuntu):
> status: New ? Triaged

...let's roll.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-01 04:52 EDT-------
fix successfully verified, 300 cycles under z/VM and in LPAR.
kernel 4.4.0-16, multipath version 0.5.0+git1.656f8865-5ubuntu1.
Closed.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-03-16 13:19 EDT-------
Again successfully verified with a lot of more zfcp fixes on Ubuntu 16.04.2. kernel 4.4.0-66-generic.