[FFE ] Merge multipath-tools from Debian unstable for noble

[Bryce Harrington]

[ FFE ]

This is an FFE to update the Ubuntu multipath-tools package from version 0.9.4-5ubuntu3 to the updated Debian 0.9.7-6. This is an upstream micro release update.

The merge was already mostly done, except with a slightly outdated debian version (0.9.7-4) which is not missing many changes so I am not concerned about that, but would like to get the paperwork done and approved before I re-do the merge process once again.

The package will undergo the following testing in addition to the autopkgtest:
Builds in Launchpad
In a VM: Upgrades from the Mantic version without failures
In a VM: Installs without failures
Manually test multipath-tools-boot with kvm

Ultimately I do not think this update will break other packages, and once the manual testing is done, will be safe to upload.

[ Changes ]

I will first show the debian changes because I think those are more interesting. The upstream changes have quite a few commits, but most were bug fixes or formatting things. I added notes to ones that I thought could be interesting, but ultimately I don’t think any are concerning.

I will also be adding a change in d/NEWS saying to ignore the socket based activation. This is coming from a debian MR https://salsa.debian.org/linux-blocks-team/multipath-tools/-/merge_requests/13

Debian changes from 0.9.4-5 -> 0.9.7-6

85ca0598 Switch Build-Depends pkg-config to pkgconf
aff95b91 Update changelog for 0.9.7-5 release
902a13b2 Rely on newly un-broken dmsetup
This adds the following constraint: dmsetup (>= 2:1.02.196-1~) which we unfortunately do not have, so we will have to patch this out for noble and keep the other patch. This shouldn’t make any issues because the patch 0002-11-dm-mpath-fix-DM_UDEV_RULES_VSN-check.patch has existed for a while.
188df312 (tag: debian/0.9.7-4) Update changelog for 0.9.7-4 release
2faddfc3 Fix initramfs test
27dfd793 (tag: debian/0.9.7-3) Update changelog for 0.9.7-3 release
cf173e59 Update changelog for 0.9.7-3 release
b7f0988c Switch Build-Depends to systemd-dev
8df9b73a (tag: debian/0.9.7-2) Update changelog for 0.9.7-2 release
018f4360 Correctly apply Debian-specific udev rule priorities
Switches 56-multipath.rules to 60-multipath.rules
Switches 60-kpartx.rules to 66-kpartx.rules
fixes new “initramfs” dep8 test
3e14ec9d (tag: debian/0.9.7-1) Update changelog for 0.9.7-1 release
11f09a78 postinst: ignore failure from udevadm
5ec6ee3e autopkgtest: add extra details, journal output
46cd7feb d/rules: drop SCSI_DH_MODULES_PRELOAD which did not work anyway
462750d0 Follow upstream change from modules-load to modprobe@ service
Removes Debian’s 0005-multipathd.service-re-add-ExecStartPre.patch

97e97823 Build-Depend: libmount-dev, now needed
ed5de582 Refresh patches
1c2d4ec7 Update upstream source from tag 'upstream/0.9.7'
232dd6eb (tag: upstream/0.9.7, origin/upstream) New upstream version 0.9.7
be45ef74 (tag: debian/0.9.4-10) Update changelog for 0.9.4-10 release
704cf13e autopkgtest: fix under upstream default of find_multipaths strict
Provides WWID entry for the autopkgtest
ff4b0637 Fix udev rules install location
Installes into /usr/lib/udev/rules.d/ instead of /usr/lib/udev/
583154b8 (tag: debian/0.9.4-9) Update changelog for 0.9.4-9 release
a42a18bb Add new autopkgtest for initramfs hooks
265cc4d9 Apply wrap-and-sort -kast
Cosmetic change to d/control
13d5c3e4 Refresh patches
bdcbd1e0 Stop installing multipath-tools-boot.service mask
d41422b1 initramfs: use modern helpers, tidy up
5598a12e initramfs: use /usr-merged layout
6c4f974c kpartx-udeb: install udev rules and helper
05a2082d Use dh_install to install udev rules
c785707a Rearrange debian/tmp usage to get more out of dh_missng
6a71dd82 (tag: debian/0.9.4-8) Update changelog for 0.9.4-8 release
dfcb2a26 Install into /usr-merged layout
b4a1e2ca (tag: debian/0.9.4-7) Update changelog for 0.9.4-7 release
bc3bacba Add NEWS for updated boot, removed sysvinit scripts
475fd458 Drop obsolete Depends: lsb-base
918d442d Remove init script, following Ubuntu bug reports
Yay systemd only now!
bdc80b97 Merge initramfs integration from Ubuntu
Yay ubuntu deltas being dropped!
There is a single delta remaining from this, and that is Debian only has the kpartx package whereas we have the kpartx-boot package, so we still need to drop kpartx.rules from the udev rules.
b9673123 (tag: debian/0.9.4-6) Update changelog for 0.9.4-6 release
857edc08 Merge branch 'real-socket-activation' into 'master'
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.
63908850 d/NEWS: Add entry about socket-activation being now the default.
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.
634733b6 Implement real socket activation of multipathd.service.
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.
818e429c d/p/0007-multipathd.service-remove-Also-multipathd.socket.patch:
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.

Upstream changes from 0.9.4 -> 0.9.7

6bb73 Merge pull request #77 from openSUSE/queue
bbb74 libmultipath: bump version to 0.9.7
225ba multipath-tools: set usr_prefix to /usr in default configuration
2ab95 libmultipath.version: bump ABI version to 22.0.0
981b8 multipathd: Add auto_resize config option
This is a new feature, but it is set to disabled by default.
8aac8 multipathd: move resize_map() to multipathd/main.c
2d465 multipathd: move post-reloading commands into resize_map()
1f980 multipathd: check and update all paths when in cli_resize
25208 libmutipath: Retain device size if sysfs_get_size fails.
a4e32 libmultipath: Add max_retries config option
b7ddf README.md: fix formatting of Changelog section
d3e23 README.md: Extend the section about NVMe
37441 README.md: update mailing list and contributing information
f5769 Makefile.inc, README.md: fix docs for prefix in split-usr case
49b0e multipath.conf.5: fix typo
6fad1 libmpathutil: remove systemd_service_enabled()
187f3 multipathd.service: require modprobe@dm_multipath.service if available
12f7d libmultipath: io_err_stat: fix error handling
bdeba libmultipath: io_err_stat: use higher number of aio slots
26dab libmultipath: io_err_stat: call io_destroy() inside free_io_err_pathvec()
de5c3 libmultipath: io_err_stat: don't free aio memory before completion
5bf3a libmultipath: directio: fix error handling
77e4e libmultipath: directio: don't reset ct->running after io_cancel()
49c95 libmultipath: reduce log level of directio messages
d563e libmpathutil: Remove parse_prkey symbol
1cada multipathd: Added support to handle FPIN-Li events for FC-NVMe
7a108 multipath-tools: fix spelling
56a82 libmultipath: avoid -Warray-bounds error in uatomic operations
ff421 libmultipath: deprecate bindings_file, wwids_file, prkeys_file
58ad9 multipath-tools: allow prefixes with and w/o trailing slash
9a21c multipath: add a missing newline
d47a6 libmultipath: print built-in values for deprecated options
39d56 multipath-tools: README.md: improve documentation for compile-time options
3948e libdmmp/Makefile: fix bug in install section
0fb7f multipath-tools man pages: generate with correct paths
6d0f9 multipath-tools: add compile time configuration for "/etc/multipath"
98236 multipath-tools Makefile: sanitize paths for configuration files
73d37 multipath-tools tests: mock pthread_mutex_{lock,unlock}
7eaeb multipathd: watch bindings file with inotify + timestamp
bb068 multipath-tools tests: add test for ordering of bindings
33e04 multipath-tools tests: adapt alias tests for total ordering
17ff9 libmultipath: simplify get_free_id() assuming total ordering
c7bd1 multipath-tools tests: fix alias test after sort order change
093b6 libmultipath: sort aliases by length and strcmp
1de72 libmultipath: adapt to new semantics of dm_get_uuid()
c02fd libmultipath: dm_get_uuid(): return emtpy UUID for non-existing maps
f51c2 multipath-tools tests: fix alias tests
e2d4f libmultipath: keep bindings in memory
a591f libmultipath: alias.c: factor out read_binding()
6b41f libmultipath: update_bindings_file: don't log temp file name
6599d libmultipath: update_bindings_file: use a single write()
5e9fc libmultipath: update_bindings_file: take filename argument
bdec0 libmultipath: alias.c: move bindings related code up
d41f4 libmultipath: rename fix_bindings_file() to update_bindings_file()
93094 libmultipath: add global variable for current bindings
25e5b multipath-tools test: use mock_bindings_file() consistently
ce437 multipath-tools tests: convert mock_{failed,used}_alias to macros
0ad8a multipath-tools test: consistent use of macros in alias test
92e9f multipath-tools tests: add tests for get_user_friendly_alias()
a5d85 multipath-tools test: simplify debugging for condlog mismatch
f6be4 libmultipath: lookup_binding: add comment about the algorithm
34404 libmultipath: never allocate an alias that's already taken
a5aa0 libmultipath: unify use_existing_alias() and get_user_friendly_alias()
1bca0 libmultipath: add alias_already_taken()
cc63b libmultipath: sysfs_set_scsi_tmo: do nothing for ACT_DRY_RUN
f3004 Merge pull request #68 from openSUSE/queue
5fb0c Fix hwtable test after "libmultipath: don't bother to recheck timeout"
df4da libmultipath: standardize datacore prioritizer timeouts
ea914 libmultipath: make prioritizer timeouts work like checker timeouts
1a9d3 libmultipath: make checker_timeout a path variable
7f108 libmultipath: don't bother to recheck timeout
fc9c4 libmultipath: bump version to 0.9.6
bbb77 libmultipath: fix max_sectors_kb on adding path
bd720 multipath-tools: treat disable_changed_wwids like other deprecated keywords
f220a multipath-tools: fix docs
c2c08 GitHub workflows: switch to Ubuntu 22.04 runner
ccc4a GitHub workflows: use Fedora 37 in native.yaml
95323 GitHub workflows: enable Debian "bookworm"
c6860 libmultipath: ignore nvme devices if nvme native multipath is enabled
058b5 libmultipath: fix dev_loss_tmo even if not set in configuration
44f1f libmultipath: dm_get_maps(): remove spurious assignment
8ed96 multipath-tools build: accept KBUILD_BUILD_TIMESTAMP from env
cc7d7 multipath-tools build: rename PKGCONFIG to PKG_CONFIG
d4697 multipath-tools: fix syntax and spelling errors
85bbf multipathd: don't assume mpp->paths will exist in need_switch_pathgroup
18975 multipathd: reload map if the path groups are out of order
6ccd7 multipathd: only refresh priorities in update_prio()
31a31 multipath-tools tests: add tests to verify PRIO_UNDEF changes
fb92d libmultipath: don't count PRIO_UNDEF paths for pathgroup priority
aa5a2 libmultipath: add "detect_pgpolicy_use_tpg" config option
f6566 libmultipath: add "detect_pgpolicy" config option
7ed58 multipath-tools tests: add tests for group_by_tpg policy
c9bdf libmultipath: add ALUA tpg path wildcard
3be31 libmultipath: don't copy pgpolicy string in get_pgpolicy_name
1dd05 libmultipath: add group_by_tpg path_grouping_policy
1d3b2 multipath-tools: fix spelling
ea733 multipath-tools: adapt HITACHI/OPEN- config to work with alua and multibus
57ee6 dm-parts.rules: fix warning reported by udevadm verify
787b6 11-dm-mpath.rules: fix warnings reported by udevadm verify
24bc5 Merge pull request #64 from openSUSE/queue
736c0 libmultipath: bump version to 0.9.5
74f54 GitHub workflows (basic): remove Ubuntu 18.04, add 22.04
71448 libmultipath: use directio checker for LIO targets
53b21 multipathd.service: remove "Also=multipathd.socket"
This is a framework to enable socket based activation, but unfortunately that feature is not making it into this release. Hopefully in the future, but there’s some issues with making it happen.
169ed libmultipathd: Avoid parsing errors due to unsupported designators
3707e multipath-tools tests: fix quoting of CFLAGS in Makefile
f6a54 multipath-tools Makefiles: handle dependency files correctly
be7f0 libmultipath.version: bump symbol version of get_uid
694a2 libmultipath: limit paths that can get wwid from environment
5e95a multipath-tools tests: use assert_true/false instead of assert()
4b83c libmultipath: reset nr_timeouts if we freed the context
fbe59 libmultipath: set init failure message when init fails
65366 libmultipath: return 'pending' state when port is in transition
51eb6 multipath.rules: fix "smart" bug with failed valid path check
b10a8 libmpathpersist: fix resource leak in update_map_pr()
a6c07 libmultipath: keep renames from stopping other multipath actions
e56a8 libmultipath: cleanup ACT_CREATE code in select_action
5d6fa libmultipath: select resize action even if reload is forced
16249 libmultipath: use select_reload_action in select_action
73062 libmultipath: bump ABI version to 18.0.0
88d46 libmultipath: pathinfo: don't fail for devices lacking INQUIRY properties
fa315 libmpathpersist: use conf->timeout for updating persistent reservations
15942 libmultipath: is_path_valid(): check if device is in use
8c52f libmultipath: orphan paths if coalesce_paths frees newmp
a658a libmultipath: skip extra vector work in remove_maps
57d50 multipathd: add missing newline to cli_del_map reply
50e2c multipathd: handle no active paths in update_map_pr
6b811 libmultipath: make prflag an enum
8d64b multipathd: make pr registration consistent

[Mitchell Dzurick (mitchdz)]

This merge is being put on hold. Let's wait until socket based activation is in upstream and then pulled into Debian. There is a PR[0] from sergio right now to address issues with socket based activation.

[0] - https://github.com/opensvc/multipath-tools/pull/78

[Chris Hofstaedtler (zeha)]

Please on the next sync/merge with Debian drop the delta to change kpartx.rules prio to 95. It appears this is useless since 2007.

Some history digging tells me:
- https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/hardy/multipath-tools/hardy/revision/18 added a modprobe call in multipath.rules
- https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/hardy/multipath-tools/hardy/revision/20 (6 months later) drops that modprobe call, but also changes the prio to 95.

So IMO this prio change was never correct, and is only a nuisance for things like the systemd autopkgtests which have to understand these distro differences.

[Mitchell Dzurick (mitchdz)]

Picking up this merge again, Debian package has made some changes that will be good to get in.

Thanks for your comment Chris! I'll make sure to address that patch during this merge.

[Mitchell Dzurick (mitchdz)]

Just want to update this bug that I have finished doing an initial merge, and dropped a ton of deltas we hold from debian (Thanks Chris!).

Some drops I am hesitant to drop such as removing kpartx rules from initramfs, I can't particularly pinpoint why we were doing that, but it was grouped up in a patch titled "multipath initramfs fixes for booting from multipathed devices" So maybe I just test booting from a multipathd device and if that works then it's fine to drop? I'm also thinking it could be related to how Ubuntu splits kpartx into kpartx-boot, but it's not part of the same commit so I'm not sure.

Either way, the current autopkgtest fails so it's time to investigate that.

[Mitchell Dzurick (mitchdz)]

Narrowed down the autopkgtest line.

tgtbasedmpaths is failing when the line `iscsiadm --mode session -r 1 --op new` is executed. multipathd.service seg faults due to this. Unfortunately it just seg faults silently so the logs don't really show much.

[Mitchell Dzurick (mitchdz)]

I see the autopkgtest passes[0] in the debian environment, so this is probably due to one of our deltas. Looking into it further.

[0] - https://ci.debian.net/packages/m/multipath-tools/unstable/amd64/42909838/

[Mitchell Dzurick (mitchdz)]

Hmmmm this is interesting, so I decided to check the Debian autopkgtest run in an ubuntu VM to see if this failure is occuring due to our Ubuntu deltas or not.

I set up a noble autopkgtest vm like so:

autopkgtest-buildvm-ubuntu-cloud -r noble -v --cloud-image-url http://cloud-images.ubuntu.com/daily/server
sudo mkdir -p /var/lib/adt-images/
sudo rm -f "/var/lib/adt-images/autopkgtest-noble-amd64.img"
sudo mv "autopkgtest-noble-amd64.img" /var/lib/adt-images/

I then downloaded the debian metadata from https://packages.debian.org/source/unstable/multipath-tools and built+ran the autopkgtest like so:

autopkgtest -U -s -o "dep8-multipath-tools" --test-name=tgtbasedmpaths ../multipath-tools_0.9.7-4.dsc -- qemu "/var/lib/adt-images/autopkgtest-noble-amd64.img"

And am seeing the same exact failure.

[Mitchell Dzurick (mitchdz)]

https://launchpad.net/bugs/2034471 Is also an issue in the new Noble package. Very easy to reproduce. Ideally I'd like to fix this, but would prefer the fix to be in Debian so we don't hold this type of delta.

$ lxc launch ubuntu-daily:noble n-vm --vm
$ lxc shell n-vm

# apt install -y lsscsi multipath-tools open-iscsi tgt
# add-apt-repository ppa:mitchdz/merge-lp2040360-noble-multipath-tools -y
# apt update -y && apt upgrade -y
# dpkg -s multipath-tools | grep Version:
Version: 0.9.7-4ubuntu1~noble1

# systemctl status multipathd.{service,socket} | grep Active -B2
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-02-20 21:56:26 UTC; 2min 6s ago
--
○ multipathd.socket - multipathd control socket
     Loaded: loaded (/usr/lib/systemd/system/multipathd.socket; enabled; preset: enabled)
     Active: inactive (dead) since Tue 2024-02-20 21:56:26 UTC; 2min 6s ago

[Utkarsh Gupta (utkarsh)]

Hey Mitch! Thanks for filing an FFe. Before I can formally approve it, I have some questions:

902a13b2 Rely on newly un-broken dmsetup
This adds the following constraint: dmsetup (>= 2:1.02.196-1~) which we unfortunately do not have, so we will have to patch this out for noble and keep the other patch. This shouldn’t make any issues because the patch 0002-11-dm-mpath-fix-DM_UDEV_RULES_VSN-check.patch has existed for a while.

Does that mean we'll revert that commit in Ubuntu? Will there be consequences? Will you test it to make sure this doesn't regress somehow with other changes in?

857edc08 Merge branch 'real-socket-activation' into 'master'
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.
63908850 d/NEWS: Add entry about socket-activation being now the default.
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.
634733b6 Implement real socket activation of multipathd.service.
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.
818e429c d/p/0007-multipathd.service-remove-Also-multipathd.socket.patch:
This change was dropped in debian, this is a change we tried to get working but unfortunately could not finish.

What does that mean in practice? We'll carry this as a delta? Why does it not work as-is, though? And will there be regressions or any consequences because of the above? Did you ensure there's no bug reports in Debian with the above changes and that there's enough smoke test coverage to ensure that everything's working as intended?

I mostly ask this because multipath-tools is an important and a key package and is widely used. 0.9.4 -> 0.9.7 looks fine, these micro-releases are also pretty stable. But I am just concerned the differences b/w Debian and Ubuntu might have some consequences or something. Could you confirm that those changes will be actively tested and hopefully won't regress on our side? :)

[Mitchell Dzurick (mitchdz)]

Hi Utkarsh!

902a13b2 Rely on newly un-broken dmsetup
There was a patch in Debian[0] to handle a mismatched variable name, which was finally updated >= 2:1.02.196-1~, so the patch was removed and now multipath-tools relies on that version.

Here we unfortunately have a slightly older version[1] 2:1.02.185-2ubuntu1 so I will make a commit to allow our older version and keep the patch from debian active[0], making a note for OO cycle that it should be safe to drop as long as the version of dmsetup by that time has this version name updated.

Digging into this further, you can see the patch in noble[2] that shows the environment variable being present, and the commit in debian adding it[3]. The latest version in debian[4] does not have this change patched in, so when this newer debian version gets pulled we will be able to drop this delta.

857edc08 Merge branch 'real-socket-activation' into 'master'
Apologies, I explained this weird. These were commits that we made in order to try and achieve socket based activation. The plan unfortunately did not go through so these changes got reverted in a later commit. Therefore, these changes are not present in the final merge, but they exist in the git commit history, so I wanted to touch on them to explain that they can be ignored. This is why I will also add a section to debian/NEWS explaining that socket based activation is not enabled for this release.

[0] - https://git.launchpad.net/ubuntu/+source/multipath-tools/tree/debian/patches/0007-11-dm-mpath-fix-DM_UDEV_RULES_VSN-check.patch?h=ubuntu/mantic-devel
[1] - https://launchpad.net/ubuntu/noble/+package/dmsetup
[2] - https://git.launchpad.net/ubuntu/+source/lvm2/tree/debian/patches/debian-changes?h=ubuntu/noble-devel
[3] - https://salsa.debian.org/lvm-team/lvm2/-/commit/1c5bf716af880fb5b34a160dcd7b4ed27f927590
[4] - https://salsa.debian.org/lvm-team/lvm2/-/blob/main/udev/10-dm.rules.in?ref_type=heads

[Mitchell Dzurick (mitchdz)]

Actually some of our changes[0] to try and enable socket based activation are still present. Let me do a little digging and confirm these changes are still good to have. I thought I recalled these changes being dropped.

I think these changes are still fine to bring in, but let me do a little investigation to be thorough.

[0] - https://salsa.debian.org/linux-blocks-team/multipath-tools/-/commit/634733b6cd526f6f79e285e5d93fc76021782ea0

[Mitchell Dzurick (mitchdz)]

Yeah I confirmed with Sergio - some of the changes are still present, but it should not be harmful. Thankfully these changes are easy to test, and as long as the service is enabled and working at boot, and after a package installation it will be fine.

I will manually test booting from a multipathd device using KVM.

[Utkarsh Gupta (utkarsh)]

OK, thanks. Looks good. Approved. Feel free to go ahead with the upload.

[Mitchell Dzurick (mitchdz)]

Thanks Utkarsh! I will re-do the merge today if I have time, but most likely tomorrow.

[Mitchell Dzurick (mitchdz)]

This merge may want to be not done for Noble. After doing the merge again I noticed a seg fault in multipathd when starting the service during my testing. This is easily repeatable in Ubuntu by uninstalling multipath-tools and then doing a fresh install. This is not reproducible in Debian during my testing.

I can root-cause this seg fault, which I still plan to do, but I feel uncomfortable so close to beta release troubleshooting a bug like this. I fear not enough testing will be able to happen.

[Mitchell Dzurick (mitchdz)]

Looking at upstream, there's not a ton that I worry about missing in noble. The only one is a fix for a memory leak that I think would make sense to cherrypick[0].

Otherwise, I don't think it is worth the risk to upgrade the package this late in the cycle, and instead effort would be better utilized to work on helping debian for the next ubuntu release on things such as the service/socket systemd interactions that I've put on hold[1].

Any objections to not doing the merge for this cycle? Or should I prioritize testing the package and try to squeeze it in?

[0] - https://github.com/openSUSE/multipath-tools/commit/b10a8fbd9da42f81687516cd345fd1ceeb60921f
[1] - https://salsa.debian.org/linux-blocks-team/multipath-tools/-/merge_requests/14

[Mitchell Dzurick (mitchdz)]

Plus, there are changes to how the systemd units interact that is a work in progress, which I think makes more sense for a **.10 release than an LTS.

[Mitchell Dzurick (mitchdz)]

Just for completeness - I mentioned a multipathd seg fault. There is no bug report for this as I could not reproduce this in Debian, so it seems Ubuntu specific. It can be easily reproduced by uninstalling multipath-tools and reinstalling it with apt once you do the merge.