Trusty: failure to detect device with 0.4.9-3ubuntu7.5

[Amit Oren]

When trying to detect a device with the latest 0.4.9-3ubuntu7.14 version, no device is detected.
When downgrading to 0.4.9-3ubuntu7 everything works fine.
See this console log for more detail: https://gist.github.com/amito/4108cd7ee895baecc9a6412e7c5fdeac

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Please could you confirm if the regression was specifically in 0.4.9-3ubuntu7.13 -> 0.4.9-3ubuntu7.14, or if it happened in an earlier update, which update that was?

You can get earlier binaries from https://launchpad.net/ubuntu/+source/multipath-tools/+publishinghistory. Look for Target=Trusty, Pocket=updates, click on the version and then your architecture and look under "Built Files".

[Louis Bouchard (louis)]

Hello,

Could you please make sure that you update the initrd.img file and reboot the server (if possible) ?

$ sudo update-initramfs -u

A sensible amount of modifications (63 separate patches) have been applied between version 0.4.9-3ubuntu7 and 0.4.9-3ubuntu7.14 on trusty, such as this one :

  * Allow device detection all through the initramfs: run multipathd instead
    of only scanning once for devices, so those that come up slower can still
    be used as a root device (LP: #1526984)

Please let us know if regeneration of the initrd.img file and a reboot fixes the issue that you are encountering.

[Amit Oren (amito)]

Thanks for the replies.

@racb After further bisection I can now say the first version the issue occurs in is 0.4.9-3ubuntu7.6.

@louis-bouchard Unfortunately, updating the initrd.img file and rebooting the server did not fix the issue.

[Louis Bouchard (louis)]

Hello,

Then the problem is not a regression introduced in 0.4.9-3ubuntu7.14.

The only difference between 7.5 and 7.6 is :

  * debian/patches/0043-alloc-keywords.patch: fix uninitialized keywords
    vector for configuration parsing for when there is no multipath.conf file
    present.

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 06 Oct 2015 22:13:59 -0400

You might want to investigate the configuration differences in the case where you are not using a multipath.conf file.

[Amit Oren (amito)]

I made a mistake in the previous comment: I re-validated it a few times on my setup, the issue is present when upgrading from 7.4 to the proposed 7.5.

[Christian Ehrhardt  (paelzer)]

That change could more be a source of what you describe.
It contains a huge set of backports in regard to ignore "local non-mpath devices"

 * Cherry-picked patches to ignore local non-mpath devices
    - 0001-multipath-add-checker_timeout-default-config-option.patch
    - 0002-Make-params-variable-local.patch
    - 0003-libmultipath-Fix-possible-string-overflow.patch
    - 0004-Update-hwtable-factorization.patch
    - 0005-Fixup-strip-trailing-whitespaces-for-getuid-return-v.patch
    - 0006-Remove-sysfs_attr-cache.patch
    - 0007-Move-setup_thread_attr-to-uevent.c.patch
    - 0008-Use-lists-for-uevent-processing.patch
    - 0009-Start-uevent-service-handler-from-main-thread.patch
    - 0010-libmultipath-rework-sysfs-handling.patch
    - 0011-Rework-sysfs-device-handling-in-multipathd.patch
    - 0012-Only-check-offline-status-for-SCSI-devices.patch
    - 0013-Check-for-offline-path-in-get_prio.patch
    - 0014-libmultipath-Remove-duplicate-calls-to-path_offline.patch
    - 0015-Update-dev_loss_tmo-for-no_path_retry.patch
    - 0016-Reload-map-for-device-read-only-setting-changes.patch
    - 0017-multipath-get-right-sysfs-value-for-checker_timeout.patch
    - 0018-multipath-handle-offlined-paths.patch
    - 0019-multipath-fix-scsi-timeout-code.patch
    - 0020-multipath-make-tgt_node_name-work-for-iscsi-devices.patch
    - 0021-multipath-cleanup-dev_loss_tmo-issues.patch
    - 0022-Fix-for-setting-0-to-fast_io_fail.patch
    - 0023-Fix-fast_io_fail-capping.patch
    - 0024-multipath-enable-getting-uevents-through-libudev.patch
    - 0025-Use-devpath-as-argument-for-sysfs-functions.patch
    - 0026-multipathd-remove-references-to-sysfs_device.patch
    - 0027-multipathd-use-struct-path-as-argument-for-event-pro.patch
    - 0028-Add-global-udev-reference-pointer-to-config.patch
    - 0029-Use-udev-enumeration-during-discovery.patch
    - 0030-use-struct-udev_device-during-discovery.patch
    - 0031-More-debugging-output-when-synchronizing-path-states.patch
    - 0032-Use-struct-udev_device-instead-of-sysdev.patch
    - 0033-discovery-Fixup-cciss-discovery.patch
    - 0035-Use-udev-devices-during-discovery.patch
    - 0036-Remove-all-references-to-hand-craftes-sysfs-code.patch ...

[Amit Oren (amito)]

Thanks.

I want to perform a git bisection on this revision.
I cloned the relevant bazaar branch (trusty-updated) but revision 0.4.9-3ubuntu7.5 does not exist there (neither in any other trusty branch). Any way in which I can get it?

[Louis Bouchard (louis)]

Hello,
The patches added between those two versions are not maintained in git but are quilt patches applied above the same source code found in multipath-tools_0.4.9.orig.tar.gz.

You could do something similar to a bisection by following these steps :

# sudo apt-get install ubuntu-dev-tools devscripts
# sudo apt-get build-dep multipath-tools
# mkdir bisect
# cd bisect
# pull-lp-source multipath-tools 0.4.9-3ubuntu7.4
# pull-lp-source -d multipath-tools 0.4.9-3ubuntu7.5
# debdiff *dsc > newpatches.patch
# cd multipath-tools-0.4.9
# patch -p1 < ../newpatches.patch
# quilt pop -a
# quilt push -a
# quilt pop 41

This will bring your source package to the patch level of 0.4.9-3ubuntu7.4. You can run the following to test that the created package does work : build the binary packages with the current set of quilt patches applied.

# dch -rm
and add a suffix to the version number using the ~ marker. This will make sure that the version is lower than the 7.5 version in the archive

multipath-tools (0.4.9-3ubuntu7.5~test1) trusty; urgency=medium
# sudo fakeroot debian/rules binary

Install the packages with the 0.4.9-3ubuntu7.5~test1 version and verify that your environment still works.

Then iterate over each new patch added between 0.4.9-3ubuntu7.4 and 0.4.9-3ubuntu7.5 by doing the following :

# quilt push
# dch -rm
multipath-tools (0.4.9-3ubuntu7.5~test2) trusty; urgency=medium
# fakeroot debian/rules binary

Install & test

When you get a package that doesn't work in your environment, it means that the last patch applied is causing your failure. You can identify the patch with :

# quilt applied | tail

The last one in the list is the patch you are looking for.

You may want to use LXC containers to create your build environment as this will install a sensible amount of packages.

[Mathieu Trudel-Lapierre (cyphermox)]

Could you please attach a debug log to this bug, and tell us which devices you expect should be detected by multipath?

To generate the logs we're looking for, you should be able to use "multipath -r -v4".

[Launchpad Janitor (janitor)]

[Expired for multipath-tools (Ubuntu) because there has been no activity for 60 days.]