Multipath: upgrade multipath-tools to upstream

[bugproxy]

Ubuntu ships an older version of multipath-tools (i.e., the userspace multipath
support; kernelspace multipath support is the device-mapper's multipath target,
which is as recent as their kernel is, so not a problem).

Some test teams already hit enough issues with fixes upstream
to demonstrate that point.

Going forward, it's very interesting to ship something more recent in Ubuntu, so
to alleviate the number of bugs (likely with severity of block and ship) that we
will have to debug/understand/interlock/backport/submit/upload to get fixed...

The point is fixing multipath-tools once - in a big step - rather than N times.

Canononical currently pull multipath from Debian, which
is multipath-tools 0.5.0, and upstream is around 144 patches/commits *ahead* of.
This is more than enough fixes/reasons to move to it, rather than stay on 0.5.0.
(upstream is basically the SLES level; as <email address hidden> drives upstream nowadays)

This is why this is being opened as feature request. I am wondering if Canonical see any objection to upgrading Ubuntu's multipath packages to a more recent version?

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1455482/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Steve Langasek (vorlon)]

Tracking an upstream trunk that has not been QAed for a release is not necessarily something we want to do. Do you know if upstream has another release planned?

Are the issues that are being encountered in testing present with multipath-tools 0.5.0? Debian has 0.5.0 but Ubuntu is still at 0.4.9. We would want to update to the latest release this cycle - but not necessarily pull unreleased trunk.

[Mauricio Faria de Oliveira (mfo)]

Hi Steve,

I see the points about upstream/unreleased trunk. I'm not aware of plans/how the dm/m-t community plans for releases.

Well, Debian 0.5.0 would help to a reasonable extent.
multipath-tools 0.5.0 fixes a number of issues encountered here, and is in a better position regarding backports.

Additionally, there's a few fixes I had to submit to Debian multipath-tools 0.5.0 (and sg3-utils), which would be required too for booting from multipath.

So, I think Debian 0.5.0 is OK, and any required more-recent commits we encounter can get backports.
Thanks.

[Mathieu Trudel-Lapierre (cyphermox)]

I went ahead and asked about a timeline for a next multipath-tools release on the dm-devel mailing list. Timeline was said to be "before summer holidays".

I'll do the multipath-tools merge with Debian so we get 0.5.0, and once there is a new release we'll see if it can make it into 15.10, if it happens early enough.

[Mauricio Faria de Oliveira (mfo)]

The BTS bugs w/ the fixes I mentioned earlier:

782488
782400
782487

@mathieu-tl I'm not sure which contry the summer holidays relate to, possibly France? (I guess Christophe lives there?) Would you know? Thanks!

[Mathieu Trudel-Lapierre (cyphermox)]

No idea. I think can operate with the usual procedures: merge now with 0.5.0, and either merge or do an updated package once a new release is available, provided it happens before Feature Freeze (August 20).

I'll include patches for the above bugs as I merge multipath-tools.

[Mauricio Faria de Oliveira (mfo)]

Heheh, alright.
Thank you.

[Mauricio Faria de Oliveira (mfo)]

Mathieu,

One of the things where Debian differs from upstream (but not new) is to stick with mpath[0-9]+ rather than mpath[a-z]+.
This is in patch 0002, iirc.
Can we switch over or (easier) rebase/move it toward the end of the series?

If it's not possible to switch over (we already know where to fix things for d-i), I'd ask to at least move that patch to the end of the series, and rebase it when required.
It's currently an inconvenient when backporting patches from upstream (many of which are rooted in a change that it complicates to do).
I believe the work to make the change is worth it / lesser than that work for backporting due to keeping it.

[Mathieu Trudel-Lapierre (cyphermox)]

Ideal for that would be to convince the Debian maintainers to drop the patch.

[Mauricio Faria de Oliveira (mfo)]

The 2 options are drop or rebase.

Regarding drop, the reason to ask Ubuntu is because Debian just released, and it would break w/ their release contents, which adds more work in updates/maintainence. AFAIK, Ubuntu releases are more independent of each other in the updates sense (please correct me if that's wrong).

Regarding rebase, definitely ideal and no problems expected w/ Debian. Ritesh (maintainer) has been friendly and reasonable w/ patch submission/discussion, and the constraint there is upstream-first, for which a downstream-patch rebase is not a problem, and actually helps w/ the backport work.

I'll follow up w/ Ritesh in a BTS bug, so that may eventually stream down to W.
Thanks.

[Mauricio Faria de Oliveira (mfo)]

Closing the mpath[0-9]/[a-z] discussion..

Just the merge with Debian will do it, as long as Ubuntu drops patch 0002.
Debian returned to mpath[a-z] (dropping patch 0002) a few months after Ubuntu branched off from 0.4.9-3 for Precise.

This will require changes to the installer packages; the disk-partition separator stuff can be removed (shouldn't be necessary with mpath[a-z]).

[Mathieu Trudel-Lapierre (cyphermox)]

I pulled both multipath-tools directly from Debian and did a merge of multipath-tools 0.5.0 with Ubuntu changes; both seem to currently be unable to notice that QEMU harddrives should be multipathed. I'm investigating the situation.

[Mauricio Faria de Oliveira (mfo)]

@mathieu-tl

> unable to notice that QEMU harddrives should be multipathed. I'm

This brings 2 things to mind:
1) the property whitelist support (see one or two of those BTS bugs, for the uid / udev ID_SERIAL change -- one is for sg3-utils)
2) (don't think this is the case) if you're not using scsi disks (say, virtio disks), those are usually not multipath'able due to the lack of SCSI ID.

Hope this helps.

[Mathieu Trudel-Lapierre (cyphermox)]

Well, that change was made for a good reason; multipath now no longer depends on scsi_id to be able to pick drives to use to multipath, and we can explicitly whitelist (or use blacklist_exclusions) devices or re-enable ID_SERIAL by modifying multipath.conf. Is this affecting other drives than the QEMU ones for your use-case?

There's still another issue though; in the installer we don't ship path selectors other than round-robin, so even if ID_SERIAL is whitelisted, you still need to modify multipath.conf to pick the round-robin selector instead of service-time -- which is now the default. I'll work on shipping service-time instead of round-robin, but the other option is to change multipath's default.

[Mauricio Faria de Oliveira (mfo)]

> Well, that change was made for a good reason; <snip>
> Is this affecting other drives than the QEMU ones for your use-case?

Agree, good reasons. Yes, the IPR disks show up as SCSI ones.
And if there's no other attributes (say, the SCSI_* udev attributes which sg3-utils udev rules can set), the IBM IPR disks won't be multipathed.
I didn't have a chance to check this w/ FC, but I suspect the WWN_ attributes are also defined by sg3-utils udev rules, and not somewhere else, so I guess they'd not work as well.

The BTS bugs for the sg3-utils/whitelist/IPR rev issues are listed in comment #5.

> instead of service-time -- which is now the default. I'll work on shipping

Ah, yeah, the service-time path selector.. also for the installer.
For the initramfs that is resolved w/ Debian (BTS 782363).

Thanks!

[Mathieu Trudel-Lapierre (cyphermox)]

Still working on it. With sg3-utils adapted to also ship a sg3-udeb package for use by the installer (which multipath-udeb can depend on so it's available to the installer), things appear to work more or less properly. To be able to ship this we still need the dm-service-time module available in multipath-modules (see bug 1469240), but I can patch this requirement to revert to round-robin path selector in the meantime.

That said, partman needs an update so that partman-multipath properly gets picked up. This may be another symptom of the module not being immediately available; next step is to test that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.5.0-7ubuntu1

---------------
multipath-tools (0.5.0-7ubuntu1) wily; urgency=medium

  * Merge from debian unstable, remaining changes: (LP: #1455482)
    - control:
      * Bump debhelper dependency to install udev rules to
        /lib/udev/rules.d, bump udev dependencies as well.
    - initramfs/hooks: use 95 not 60 for multipath rules priority
    - multipath-tools-boot.init: remove in favor of kpartx.udev rules (at top)
    - multipath-tools.preinst: modprobe dm-multipath.
      This will make sure that multipathd will be able to start.
    - patches/1000--set-umask-in-multipathd.patch: Set umask in multipathd.
    - rules: Move udev rules to priority 95, because rules that load
      modules should be >90.
    - debian/initramfs/local-top: wait for udev to settle before running
      'multipath' in order to avoid race condition on device-mapper calls.
    - debian/initramfs/local-top: remove '--timeout 10' which causes my
      test system to not boot roughly 3 out of 4 times.
    - Split kpartx initramfs bits into kpartx-boot for dmraid (LP: #941874)
    - Added debian/patches/0015-shared-lock-for-udev.patch (LP: #1431650)
  * debian/patches/0015-libmultipath-property-whitelist-SCSI_IDENT.patch: add
    SCSI_IDENT_* properties to blacklist exceptions, so that we can have QEMU
    multipathed devices as well as others (IBM IPR) detected properly as
    multipathed devices.
  * debian/patches/handle_spaces_in_rev_attr.patch: support IBM IPR devices
    and others which may have only spaces for the rev attribute.
  * debian/patches/path_selector.patch: switch the default path selector
    back to round-robin while service-time isn't available to the installer
    multipath-modules.
  * debian/control: add sg3-udeb to multipath-udeb Depends.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 06 Jul 2015 13:15:22 -0400

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-08-27 13:19 EDT-------
Closed at IBM side.