Activity log for bug #2026881

Date Who What changed Old value New value Message
2023-07-11 15:19:19 Arnaud Veron bug added bug
2023-07-11 15:35:35 Arnaud Veron attachment added apport file report https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2026881/+attachment/5685434/+files/apport.multipath-tools.uh_u2k4x.apport
2023-07-12 14:40:58 Sergio Durigan Junior nominated for series Ubuntu Jammy
2023-07-12 14:40:58 Sergio Durigan Junior bug task added multipath-tools (Ubuntu Jammy)
2023-07-12 14:40:58 Sergio Durigan Junior nominated for series Ubuntu Mantic
2023-07-12 14:40:58 Sergio Durigan Junior bug task added multipath-tools (Ubuntu Mantic)
2023-07-12 14:40:58 Sergio Durigan Junior nominated for series Ubuntu Lunar
2023-07-12 14:40:58 Sergio Durigan Junior bug task added multipath-tools (Ubuntu Lunar)
2023-07-12 14:47:13 Sergio Durigan Junior bug added subscriber Ubuntu Server
2023-07-12 14:47:18 Sergio Durigan Junior bug added subscriber Sergio Durigan Junior
2023-07-12 14:47:25 Sergio Durigan Junior tags server-todo
2023-07-13 18:03:54 Mitchell Dzurick multipath-tools (Ubuntu Mantic): status New In Progress
2023-07-13 18:03:56 Mitchell Dzurick multipath-tools (Ubuntu Mantic): assignee Mitchell Dzurick (mitchdz)
2023-07-13 18:03:58 Mitchell Dzurick multipath-tools (Ubuntu Lunar): status New In Progress
2023-07-13 18:04:00 Mitchell Dzurick multipath-tools (Ubuntu Lunar): assignee Mitchell Dzurick (mitchdz)
2023-07-13 18:04:02 Mitchell Dzurick multipath-tools (Ubuntu Jammy): status New In Progress
2023-07-13 18:04:05 Mitchell Dzurick multipath-tools (Ubuntu Jammy): assignee Mitchell Dzurick (mitchdz)
2023-07-13 18:42:46 Mitchell Dzurick description Hi, scsi3 pgr managed by mpathpersist command is broken in multipath-tools (0.8.8-1ubuntu1.22.04.1) 1) Release ---------- root@qau22c13n2:~# lsb_release -rd Description: Ubuntu 22.04.2 LTS Release: 22.04 2) Package version ------------------ root@qau22c13n2:~# apt-cache policy multipath-tools multipath-tools: Installed: 0.8.8-1ubuntu1.22.04.1 Candidate: 0.8.8-1ubuntu1.22.04.1 Version table: *** 0.8.8-1ubuntu1.22.04.1 500 500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages 500 http://us.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages 100 /var/lib/dpkg/status 0.8.8-1ubuntu1 500 500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages 3) What you expected to happen ------------------------------ Log from 20.04, which is not impacted by the issue root@qau20c26n1:~# mpathpersist -i -k /dev/dm-18 PR generation=0x4, 0 registered reservation key. root@qau20c26n1:~# mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-18 Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: pr message=map 36589cfc00000063203b03118b408db8c setprkey key 214872831999999 Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: message=map 36589cfc00000063203b03118b408db8c setprkey key 214872831999999 reply=ok Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: pr message=map 36589cfc00000063203b03118b408db8c setprstatus Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: message=map 36589cfc00000063203b03118b408db8c setprstatus reply=ok root@qau20c26n1:~# echo $? 0 root@qau20c26n1:~# mpathpersist -i -k /dev/dm-18 PR generation=0x6, 2 registered reservation keys follow: 0x214872831999999 0x214872831999999 => the key is correctly registered 4) What happened instead ------------------------ root@qau22c13n2:~# mpathpersist -i -k /dev/dm-14 PR generation=0x3, 0 registered reservation key. root@qau22c13n2:~# mpathpersist -v2 --out --register-ignore --param-sark=0x214872831593209 /dev/dm-14 8710.372441 | 36589cfc0000007507940a4aaf05aead6: adding new path sdv 8710.375149 | 36589cfc0000007507940a4aaf05aead6: adding new path sdac 8710.375236 | 36589cfc0000007507940a4aaf05aead6: pr message=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209 8710.375393 | 36589cfc0000007507940a4aaf05aead6: message=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209 reply=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209: Missing argument multipath-tools v0.8.8 (03/12, 2021) CLI commands reference: list|show paths list|show paths format $format list|show paths raw format $format list|show path $path list|show maps|multipaths list|show status list|show daemon list|show maps|multipaths status list|show maps|multipaths stats list|show maps|multipaths format $format list|show maps|multipaths raw format $format list|show maps|multipaths topology list|show topology list|show maps|multipaths json list|show map|multipath $map topology list|show map|multipath $map format $format list|show map|multipath $map raw format $format list|show map|multipath $map json list|show config local list|show config list|show blacklist list|show devices list|show wildcards reset maps|multipaths stats reset map|multipath $map stats add path $path remove|del path $path add map|multipath $map remove|del map|multipath $map remove|del maps|multipaths switch|switchgroup map|multipath $map group $group reconfigure reconfigure all suspend map|multipath $map resume map|multipath $map resize map|multipath $map reload map|multipath $map reset map|multipath $map reinstate path $path fail path $path disablequeueing map|multipath $map restorequeueing map|multipath $map disablequeueing maps|multipaths restorequeueing maps|multipaths quit|exit shutdown getprstatus map|multipath $map setprstatus map|multipath $map unsetprstatus map|multipath $map forcequeueing daemon restorequeueing daemon getprkey map|multipath $map setprkey map|multipath $map key $key unsetprkey map|multipath $map setmarginal path $path unsetmarginal path $path unsetmarginal map|multipath $map 8710.375427 | 36589cfc0000007507940a4aaf05aead6: failed to set prkey for multipathd. PR out: command failed root@qau22c13n2:~# echo $? 13 root@qau22c13n2:~# mpathpersist -i -k /dev/dm-14 PR generation=0x3, 0 registered reservation key. => the key is not registered 5) Root Cause ------------- multipath-tools (0.8.8-1ubuntu1.22.04.1) introduced a regression from the patch below (from CVE-2022-41974) https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 commit f812466f68b8e020818c6454d7b7a7e278bc99f6 Author: Martin Wilck <mwilck@suse.com> Date: Wed Aug 24 19:05:08 2022 +0200 multipathd: more robust command parsing Before this patch, multipathd command "map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209" was fine. With this patch, it has to be "setprkey map 36589cfc0000007507940a4aaf05aead6 key 214872831593209" It has been fixed upstream in the patch below: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 commit 04912e2fea94da2e7ccedabab5725b07ade2bcd4 Author: Benjamin Marzinski <bmarzins@redhat.com> Date: Tue Nov 29 22:56:48 2022 -0600 libmpathpersist: fix command keyword ordering Thanks for your work. [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 [Fix] A patch from upstream fixes this handle by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Case] What should happen ------------------------------ Log from 20.04, which is not impacted by the issue root@qau20c26n1:~# mpathpersist -i -k /dev/dm-18 PR generation=0x4, 0 registered reservation key. root@qau20c26n1:~# mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-18 Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: pr message=map 36589cfc00000063203b03118b408db8c setprkey key 214872831999999 Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: message=map 36589cfc00000063203b03118b408db8c setprkey key 214872831999999 reply=ok Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: pr message=map 36589cfc00000063203b03118b408db8c setprstatus Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: message=map 36589cfc00000063203b03118b408db8c setprstatus reply=ok root@qau20c26n1:~# echo $? 0 root@qau20c26n1:~# mpathpersist -i -k /dev/dm-18 PR generation=0x6, 2 registered reservation keys follow: 0x214872831999999 0x214872831999999 => the key is correctly registered What happens when a failure occurs ------------------------ root@qau22c13n2:~# mpathpersist -i -k /dev/dm-14 PR generation=0x3, 0 registered reservation key. root@qau22c13n2:~# mpathpersist -v2 --out --register-ignore --param-sark=0x214872831593209 /dev/dm-14 8710.372441 | 36589cfc0000007507940a4aaf05aead6: adding new path sdv 8710.375149 | 36589cfc0000007507940a4aaf05aead6: adding new path sdac 8710.375236 | 36589cfc0000007507940a4aaf05aead6: pr message=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209 8710.375393 | 36589cfc0000007507940a4aaf05aead6: message=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209 reply=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209: Missing argument multipath-tools v0.8.8 (03/12, 2021) CLI commands reference: list|show paths list|show paths format $format list|show paths raw format $format list|show path $path list|show maps|multipaths list|show status list|show daemon list|show maps|multipaths status list|show maps|multipaths stats list|show maps|multipaths format $format list|show maps|multipaths raw format $format list|show maps|multipaths topology list|show topology list|show maps|multipaths json list|show map|multipath $map topology list|show map|multipath $map format $format list|show map|multipath $map raw format $format list|show map|multipath $map json list|show config local list|show config list|show blacklist list|show devices list|show wildcards reset maps|multipaths stats reset map|multipath $map stats add path $path remove|del path $path add map|multipath $map remove|del map|multipath $map remove|del maps|multipaths switch|switchgroup map|multipath $map group $group reconfigure reconfigure all suspend map|multipath $map resume map|multipath $map resize map|multipath $map reload map|multipath $map reset map|multipath $map reinstate path $path fail path $path disablequeueing map|multipath $map restorequeueing map|multipath $map disablequeueing maps|multipaths restorequeueing maps|multipaths quit|exit shutdown getprstatus map|multipath $map setprstatus map|multipath $map unsetprstatus map|multipath $map forcequeueing daemon restorequeueing daemon getprkey map|multipath $map setprkey map|multipath $map key $key unsetprkey map|multipath $map setmarginal path $path unsetmarginal path $path unsetmarginal map|multipath $map 8710.375427 | 36589cfc0000007507940a4aaf05aead6: failed to set prkey for multipathd. PR out: command failed root@qau22c13n2:~# echo $? 13 root@qau22c13n2:~# mpathpersist -i -k /dev/dm-14 PR generation=0x3, 0 registered reservation key. => the key is not registered [Where things can go wrong] The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries. I looked through the commits between the one that fixes this bug and the latest commit, and did not see any patches or open issues related to this bug.
2023-07-13 18:54:14 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/446796
2023-07-13 18:59:43 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/446797
2023-07-17 17:35:43 Mitchell Dzurick merge proposal linked https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/446987
2023-07-31 22:01:14 Launchpad Janitor multipath-tools (Ubuntu Mantic): status In Progress Fix Released
2023-07-31 22:01:14 Launchpad Janitor bug watch added https://github.com/opensvc/multipath-tools/issues/26
2023-07-31 22:01:14 Launchpad Janitor cve linked 2022-41973
2023-07-31 22:01:14 Launchpad Janitor cve linked 2022-41974
2023-08-04 17:46:16 Mitchell Dzurick description [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 [Fix] A patch from upstream fixes this handle by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Case] What should happen ------------------------------ Log from 20.04, which is not impacted by the issue root@qau20c26n1:~# mpathpersist -i -k /dev/dm-18 PR generation=0x4, 0 registered reservation key. root@qau20c26n1:~# mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-18 Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: pr message=map 36589cfc00000063203b03118b408db8c setprkey key 214872831999999 Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: message=map 36589cfc00000063203b03118b408db8c setprkey key 214872831999999 reply=ok Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: pr message=map 36589cfc00000063203b03118b408db8c setprstatus Jul 11 17:10:27 | 36589cfc00000063203b03118b408db8c: message=map 36589cfc00000063203b03118b408db8c setprstatus reply=ok root@qau20c26n1:~# echo $? 0 root@qau20c26n1:~# mpathpersist -i -k /dev/dm-18 PR generation=0x6, 2 registered reservation keys follow: 0x214872831999999 0x214872831999999 => the key is correctly registered What happens when a failure occurs ------------------------ root@qau22c13n2:~# mpathpersist -i -k /dev/dm-14 PR generation=0x3, 0 registered reservation key. root@qau22c13n2:~# mpathpersist -v2 --out --register-ignore --param-sark=0x214872831593209 /dev/dm-14 8710.372441 | 36589cfc0000007507940a4aaf05aead6: adding new path sdv 8710.375149 | 36589cfc0000007507940a4aaf05aead6: adding new path sdac 8710.375236 | 36589cfc0000007507940a4aaf05aead6: pr message=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209 8710.375393 | 36589cfc0000007507940a4aaf05aead6: message=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209 reply=map 36589cfc0000007507940a4aaf05aead6 setprkey key 214872831593209: Missing argument multipath-tools v0.8.8 (03/12, 2021) CLI commands reference: list|show paths list|show paths format $format list|show paths raw format $format list|show path $path list|show maps|multipaths list|show status list|show daemon list|show maps|multipaths status list|show maps|multipaths stats list|show maps|multipaths format $format list|show maps|multipaths raw format $format list|show maps|multipaths topology list|show topology list|show maps|multipaths json list|show map|multipath $map topology list|show map|multipath $map format $format list|show map|multipath $map raw format $format list|show map|multipath $map json list|show config local list|show config list|show blacklist list|show devices list|show wildcards reset maps|multipaths stats reset map|multipath $map stats add path $path remove|del path $path add map|multipath $map remove|del map|multipath $map remove|del maps|multipaths switch|switchgroup map|multipath $map group $group reconfigure reconfigure all suspend map|multipath $map resume map|multipath $map resize map|multipath $map reload map|multipath $map reset map|multipath $map reinstate path $path fail path $path disablequeueing map|multipath $map restorequeueing map|multipath $map disablequeueing maps|multipaths restorequeueing maps|multipaths quit|exit shutdown getprstatus map|multipath $map setprstatus map|multipath $map unsetprstatus map|multipath $map forcequeueing daemon restorequeueing daemon getprkey map|multipath $map setprkey map|multipath $map key $key unsetprkey map|multipath $map setmarginal path $path unsetmarginal path $path unsetmarginal map|multipath $map 8710.375427 | 36589cfc0000007507940a4aaf05aead6: failed to set prkey for multipathd. PR out: command failed root@qau22c13n2:~# echo $? 13 root@qau22c13n2:~# mpathpersist -i -k /dev/dm-14 PR generation=0x3, 0 registered reservation key. => the key is not registered [Where things can go wrong] The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries. I looked through the commits between the one that fixes this bug and the latest commit, and did not see any patches or open issues related to this bug. [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 A patch from upstream fixes this by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Plan] The autopkgtest provide sufficient testing to spot for regressions, specifically tgtbasedmpaths does a full set-up and stress test of the multipath on local iscsi targets. Therefore, the targeted test here will be for mpathpersist and ensuring the command ordering is working, on-top of that, ensuring the multipath-tools package upgrades in-place without any problems. 1. Set up a VM 2. Login to VM, and install required packages # apt install -y lsscsi multipath-tools open-iscsi tgt fi // If testing new PPA add that here # add-apt-repository -y ppa:mitchdz/multipath-tools-2026881 && apt update -y && apt upgrade -y 3. Setup targets to test against ``` targetname="iqn.2016-11.foo.com:target.iscsi" cwd=$(pwd) testdir="/mnt/tgtmpathtest" localhost="127.0.0.1" portal="${localhost}:3260" maxpaths=4 backfn="backingfile" expectwwid="60000000000000000e00000000010001" testdisk="/dev/disk/by-id/wwn-0x${expectwwid}" ### Setup mpath devices # Restart tgtd to make sure modules are all loaded service tgt restart || echo "Failed to restart tgt" >&2 # prep SINGLE test file truncate --size 100M ${backfn} # create target tgtadm --lld iscsi --op new --mode target --tid 1 -T "${targetname}" # allow all to bind the target tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL # set backing file tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b "${cwd}/${backfn}" # scan for targets (locally) iscsiadm --mode discovery --type sendtargets --portal ${localhost} # login echo "login #1" iscsiadm --mode node --targetname "${targetname}" --portal ${portal} --login # duplicate this session (always 1) for i in $(seq 2 ${maxpaths}) do echo "extra login #${i}" iscsiadm --mode session -r 1 --op new done udevadm settle sleep 5 # sleep a bit to allow device to be created. # status summary echo "Status after initial setup" tgtadm --lld iscsi --mode target --op show tgtadm --lld iscsi --op show --mode conn --tid 1 iscsiadm --mode session -P 1 lsscsi -liv multipath -v3 -ll dmsetup table echo "Test WWN should now point to DM" readlink "${testdisk}" | grep dm ``` 4. test mpathpersist usage // Current (failing) case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 184.785211 | mpatha: adding new path sdd 184.786687 | mpatha: adding new path sda 184.788045 | mpatha: adding new path sdc 184.789458 | mpatha: adding new path sdb 184.790255 | mpatha: pr message=map mpatha setprstatus 184.790521 | mpatha: message=map mpatha setprstatus reply=map mpatha setprstatus: Missing argument .... // Fixed case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 299.345497 | mpatha: adding new path sdd 299.347042 | mpatha: adding new path sdb 299.348504 | mpatha: adding new path sdc 299.349918 | mpatha: adding new path sda 299.350760 | mpatha: pr message=setprstatus map mpatha 299.351526 | mpatha: message=setprstatus map mpatha reply=ok 5. Test in-place upgrade # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 8:0:0:1 sdc 8:32 active ready running # sudo apt install multipath-tools=0.8.8-1ubuntu1.22.04.2~jammy5 # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 8:0:0:1 sdc 8:32 active ready running [Where things can go wrong] - The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries. - I looked through the commits between the one that fixes this bug and the latest commit, and did not see any patches or open issues related to this bug.
2023-08-04 17:56:56 Mitchell Dzurick description [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 A patch from upstream fixes this by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Plan] The autopkgtest provide sufficient testing to spot for regressions, specifically tgtbasedmpaths does a full set-up and stress test of the multipath on local iscsi targets. Therefore, the targeted test here will be for mpathpersist and ensuring the command ordering is working, on-top of that, ensuring the multipath-tools package upgrades in-place without any problems. 1. Set up a VM 2. Login to VM, and install required packages # apt install -y lsscsi multipath-tools open-iscsi tgt fi // If testing new PPA add that here # add-apt-repository -y ppa:mitchdz/multipath-tools-2026881 && apt update -y && apt upgrade -y 3. Setup targets to test against ``` targetname="iqn.2016-11.foo.com:target.iscsi" cwd=$(pwd) testdir="/mnt/tgtmpathtest" localhost="127.0.0.1" portal="${localhost}:3260" maxpaths=4 backfn="backingfile" expectwwid="60000000000000000e00000000010001" testdisk="/dev/disk/by-id/wwn-0x${expectwwid}" ### Setup mpath devices # Restart tgtd to make sure modules are all loaded service tgt restart || echo "Failed to restart tgt" >&2 # prep SINGLE test file truncate --size 100M ${backfn} # create target tgtadm --lld iscsi --op new --mode target --tid 1 -T "${targetname}" # allow all to bind the target tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL # set backing file tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b "${cwd}/${backfn}" # scan for targets (locally) iscsiadm --mode discovery --type sendtargets --portal ${localhost} # login echo "login #1" iscsiadm --mode node --targetname "${targetname}" --portal ${portal} --login # duplicate this session (always 1) for i in $(seq 2 ${maxpaths}) do echo "extra login #${i}" iscsiadm --mode session -r 1 --op new done udevadm settle sleep 5 # sleep a bit to allow device to be created. # status summary echo "Status after initial setup" tgtadm --lld iscsi --mode target --op show tgtadm --lld iscsi --op show --mode conn --tid 1 iscsiadm --mode session -P 1 lsscsi -liv multipath -v3 -ll dmsetup table echo "Test WWN should now point to DM" readlink "${testdisk}" | grep dm ``` 4. test mpathpersist usage // Current (failing) case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 184.785211 | mpatha: adding new path sdd 184.786687 | mpatha: adding new path sda 184.788045 | mpatha: adding new path sdc 184.789458 | mpatha: adding new path sdb 184.790255 | mpatha: pr message=map mpatha setprstatus 184.790521 | mpatha: message=map mpatha setprstatus reply=map mpatha setprstatus: Missing argument .... // Fixed case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 299.345497 | mpatha: adding new path sdd 299.347042 | mpatha: adding new path sdb 299.348504 | mpatha: adding new path sdc 299.349918 | mpatha: adding new path sda 299.350760 | mpatha: pr message=setprstatus map mpatha 299.351526 | mpatha: message=setprstatus map mpatha reply=ok 5. Test in-place upgrade # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 8:0:0:1 sdc 8:32 active ready running # sudo apt install multipath-tools=0.8.8-1ubuntu1.22.04.2~jammy5 # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 8:0:0:1 sdc 8:32 active ready running [Where things can go wrong] - The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries. - I looked through the commits between the one that fixes this bug and the latest commit, and did not see any patches or open issues related to this bug. [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 A patch from upstream fixes this by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Plan] The autopkgtest provide sufficient testing to spot for regressions, specifically tgtbasedmpaths does a full set-up and stress test of the multipath on local iscsi targets. Therefore, the targeted test here will be for mpathpersist and ensuring the command ordering is working, on-top of that, ensuring the multipath-tools package upgrades in-place without any problems. 1. Set up a VM 2. Login to VM, and install required packages # apt install -y lsscsi multipath-tools open-iscsi tgt // If testing new PPA add that here # add-apt-repository -y ppa:mitchdz/multipath-tools-2026881 && apt update -y && apt upgrade -y 3. Setup targets to test against ``` targetname="iqn.2016-11.foo.com:target.iscsi" cwd=$(pwd) testdir="/mnt/tgtmpathtest" localhost="127.0.0.1" portal="${localhost}:3260" maxpaths=4 backfn="backingfile" expectwwid="60000000000000000e00000000010001" testdisk="/dev/disk/by-id/wwn-0x${expectwwid}" ### Setup mpath devices # Restart tgtd to make sure modules are all loaded service tgt restart || echo "Failed to restart tgt" >&2 # prep SINGLE test file truncate --size 100M ${backfn} # create target tgtadm --lld iscsi --op new --mode target --tid 1 -T "${targetname}" # allow all to bind the target tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL # set backing file tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b "${cwd}/${backfn}" # scan for targets (locally) iscsiadm --mode discovery --type sendtargets --portal ${localhost} # login echo "login #1" iscsiadm --mode node --targetname "${targetname}" --portal ${portal} --login # duplicate this session (always 1) for i in $(seq 2 ${maxpaths}) do     echo "extra login #${i}"     iscsiadm --mode session -r 1 --op new done udevadm settle sleep 5 # sleep a bit to allow device to be created. # status summary echo "Status after initial setup" tgtadm --lld iscsi --mode target --op show tgtadm --lld iscsi --op show --mode conn --tid 1 iscsiadm --mode session -P 1 lsscsi -liv multipath -v3 -ll dmsetup table echo "Test WWN should now point to DM" readlink "${testdisk}" | grep dm ``` 4. test mpathpersist usage // Current (failing) case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 184.785211 | mpatha: adding new path sdd 184.786687 | mpatha: adding new path sda 184.788045 | mpatha: adding new path sdc 184.789458 | mpatha: adding new path sdb 184.790255 | mpatha: pr message=map mpatha setprstatus 184.790521 | mpatha: message=map mpatha setprstatus reply=map mpatha setprstatus: Missing argument .... // Fixed case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 299.345497 | mpatha: adding new path sdd 299.347042 | mpatha: adding new path sdb 299.348504 | mpatha: adding new path sdc 299.349918 | mpatha: adding new path sda 299.350760 | mpatha: pr message=setprstatus map mpatha 299.351526 | mpatha: message=setprstatus map mpatha reply=ok 5. Test in-place upgrade # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled   `- 8:0:0:1 sdc 8:32 active ready running # sudo apt install multipath-tools=0.8.8-1ubuntu1.22.04.2~jammy5 # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled   `- 8:0:0:1 sdc 8:32 active ready running [Where things can go wrong] - The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries. - I looked through the commits between the one that fixes this bug and the latest commit, and did not see any patches or open issues related to this bug.
2023-08-09 03:43:19 Ubuntu Archive Robot bug added subscriber Bryce Harrington
2023-08-15 21:14:08 Mitchell Dzurick description [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 A patch from upstream fixes this by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Plan] The autopkgtest provide sufficient testing to spot for regressions, specifically tgtbasedmpaths does a full set-up and stress test of the multipath on local iscsi targets. Therefore, the targeted test here will be for mpathpersist and ensuring the command ordering is working, on-top of that, ensuring the multipath-tools package upgrades in-place without any problems. 1. Set up a VM 2. Login to VM, and install required packages # apt install -y lsscsi multipath-tools open-iscsi tgt // If testing new PPA add that here # add-apt-repository -y ppa:mitchdz/multipath-tools-2026881 && apt update -y && apt upgrade -y 3. Setup targets to test against ``` targetname="iqn.2016-11.foo.com:target.iscsi" cwd=$(pwd) testdir="/mnt/tgtmpathtest" localhost="127.0.0.1" portal="${localhost}:3260" maxpaths=4 backfn="backingfile" expectwwid="60000000000000000e00000000010001" testdisk="/dev/disk/by-id/wwn-0x${expectwwid}" ### Setup mpath devices # Restart tgtd to make sure modules are all loaded service tgt restart || echo "Failed to restart tgt" >&2 # prep SINGLE test file truncate --size 100M ${backfn} # create target tgtadm --lld iscsi --op new --mode target --tid 1 -T "${targetname}" # allow all to bind the target tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL # set backing file tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b "${cwd}/${backfn}" # scan for targets (locally) iscsiadm --mode discovery --type sendtargets --portal ${localhost} # login echo "login #1" iscsiadm --mode node --targetname "${targetname}" --portal ${portal} --login # duplicate this session (always 1) for i in $(seq 2 ${maxpaths}) do     echo "extra login #${i}"     iscsiadm --mode session -r 1 --op new done udevadm settle sleep 5 # sleep a bit to allow device to be created. # status summary echo "Status after initial setup" tgtadm --lld iscsi --mode target --op show tgtadm --lld iscsi --op show --mode conn --tid 1 iscsiadm --mode session -P 1 lsscsi -liv multipath -v3 -ll dmsetup table echo "Test WWN should now point to DM" readlink "${testdisk}" | grep dm ``` 4. test mpathpersist usage // Current (failing) case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 184.785211 | mpatha: adding new path sdd 184.786687 | mpatha: adding new path sda 184.788045 | mpatha: adding new path sdc 184.789458 | mpatha: adding new path sdb 184.790255 | mpatha: pr message=map mpatha setprstatus 184.790521 | mpatha: message=map mpatha setprstatus reply=map mpatha setprstatus: Missing argument .... // Fixed case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 299.345497 | mpatha: adding new path sdd 299.347042 | mpatha: adding new path sdb 299.348504 | mpatha: adding new path sdc 299.349918 | mpatha: adding new path sda 299.350760 | mpatha: pr message=setprstatus map mpatha 299.351526 | mpatha: message=setprstatus map mpatha reply=ok 5. Test in-place upgrade # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled   `- 8:0:0:1 sdc 8:32 active ready running # sudo apt install multipath-tools=0.8.8-1ubuntu1.22.04.2~jammy5 # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled   `- 8:0:0:1 sdc 8:32 active ready running [Where things can go wrong] - The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries. - I looked through the commits between the one that fixes this bug and the latest commit, and did not see any patches or open issues related to this bug. [Impact] The command ordering between libmpathpersist and multipathd is not using the correct keyword for commands. The result is when attempting to use mpathpersist a "Missing argument" is displayed and the command can not be executed properly. This is a regression caused by a CVE fix in release 0.8.8-1ubuntu1.22.04.1, https://github.com/opensvc/multipath-tools/commit/f812466f68b8e020818c6454d7b7a7e278bc99f6 A patch from upstream fixes this by handling different command ordering: https://github.com/opensvc/multipath-tools/commit/04912e2fea94da2e7ccedabab5725b07ade2bcd4 [Test Plan] The autopkgtest provide sufficient testing to spot for regressions, specifically tgtbasedmpaths does a full set-up and stress test of the multipath on local iscsi targets. Therefore, the targeted test here will be for mpathpersist and ensuring the command ordering is working, on-top of that, ensuring the multipath-tools package upgrades in-place without any problems. 1. Set up a VM 2. Login to VM, and install required packages # apt install -y lsscsi multipath-tools open-iscsi tgt // If testing new PPA add that here # add-apt-repository -y ppa:mitchdz/multipath-tools-2026881 && apt update -y && apt upgrade -y 3. Setup targets to test against ``` targetname="iqn.2016-11.foo.com:target.iscsi" cwd=$(pwd) testdir="/mnt/tgtmpathtest" localhost="127.0.0.1" portal="${localhost}:3260" maxpaths=4 backfn="backingfile" expectwwid="60000000000000000e00000000010001" testdisk="/dev/disk/by-id/wwn-0x${expectwwid}" ### Setup mpath devices # Restart tgtd to make sure modules are all loaded service tgt restart || echo "Failed to restart tgt" >&2 # prep SINGLE test file truncate --size 100M ${backfn} # create target tgtadm --lld iscsi --op new --mode target --tid 1 -T "${targetname}" # allow all to bind the target tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL # set backing file tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b "${cwd}/${backfn}" # scan for targets (locally) iscsiadm --mode discovery --type sendtargets --portal ${localhost} # login echo "login #1" iscsiadm --mode node --targetname "${targetname}" --portal ${portal} --login # duplicate this session (always 1) for i in $(seq 2 ${maxpaths}) do     echo "extra login #${i}"     iscsiadm --mode session -r 1 --op new done udevadm settle sleep 5 # sleep a bit to allow device to be created. # status summary echo "Status after initial setup" tgtadm --lld iscsi --mode target --op show tgtadm --lld iscsi --op show --mode conn --tid 1 iscsiadm --mode session -P 1 lsscsi -liv multipath -v3 -ll dmsetup table echo "Test WWN should now point to DM" readlink "${testdisk}" | grep dm ``` 4. test mpathpersist usage // Current (failing) case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 184.785211 | mpatha: adding new path sdd 184.786687 | mpatha: adding new path sda 184.788045 | mpatha: adding new path sdc 184.789458 | mpatha: adding new path sdb 184.790255 | mpatha: pr message=map mpatha setprstatus 184.790521 | mpatha: message=map mpatha setprstatus reply=map mpatha setprstatus: Missing argument .... // Fixed case # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # mpathpersist -v2 --out --register-ignore --param-sark=0x214872831999999 /dev/dm-0 299.345497 | mpatha: adding new path sdd 299.347042 | mpatha: adding new path sdb 299.348504 | mpatha: adding new path sdc 299.349918 | mpatha: adding new path sda 299.350760 | mpatha: pr message=setprstatus map mpatha 299.351526 | mpatha: message=setprstatus map mpatha reply=ok 5. Test in-place upgrade # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.1 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled   `- 8:0:0:1 sdc 8:32 active ready running # sudo apt install multipath-tools=0.8.8-1ubuntu1.22.04.2~jammy5 # dpkg -l multipath-tools | awk '/^ii/ {print $3}' 0.8.8-1ubuntu1.22.04.2~jammy5 # multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=100M features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 9:0:0:1 sdd 8:48 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 6:0:0:1 sda 8:0 active ready running |-+- policy='service-time 0' prio=1 status=enabled | `- 7:0:0:1 sdb 8:16 active ready running `-+- policy='service-time 0' prio=1 status=enabled   `- 8:0:0:1 sdc 8:32 active ready running [Where things can go wrong] - The applied patch is relatively small and handles different command ordering. A potential issue I foresee is other occurences of a mismatch between the libaries.
2023-08-18 06:22:45 Timo Aaltonen multipath-tools (Ubuntu Lunar): status In Progress Fix Committed
2023-08-18 06:22:46 Timo Aaltonen bug added subscriber Ubuntu Stable Release Updates Team
2023-08-18 06:22:47 Timo Aaltonen bug added subscriber SRU Verification
2023-08-18 06:22:50 Timo Aaltonen tags server-todo server-todo verification-needed verification-needed-lunar
2023-08-18 06:28:47 Timo Aaltonen multipath-tools (Ubuntu Jammy): status In Progress Fix Committed
2023-08-18 06:28:51 Timo Aaltonen tags server-todo verification-needed verification-needed-lunar server-todo verification-needed verification-needed-jammy verification-needed-lunar
2023-08-22 06:26:47 Arnaud Veron tags server-todo verification-needed verification-needed-jammy verification-needed-lunar server-todo verification-done-jammy verification-needed verification-needed-lunar
2023-08-29 22:19:27 Mitchell Dzurick tags server-todo verification-done-jammy verification-needed verification-needed-lunar server-todo verification-done-jammy verification-doneverification-done-lunar
2023-08-30 15:16:22 Robie Basak tags server-todo verification-done-jammy verification-doneverification-done-lunar server-todo verification-done verification-done-jammy verification-done-lunar
2023-09-20 13:10:57 Launchpad Janitor merge proposal linked https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/451737
2023-09-20 13:28:36 Mitchell Dzurick merge proposal unlinked https://code.launchpad.net/~mitchdz/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/451737
2023-10-06 21:47:55 Ubuntu Archive Robot bug added subscriber Andreas Hasenack
2023-10-13 11:36:54 Timo Aaltonen tags server-todo verification-done verification-done-jammy verification-done-lunar server-todo verification-done-jammy verification-needed verification-needed-lunar
2023-10-13 11:38:08 Timo Aaltonen tags server-todo verification-done-jammy verification-needed verification-needed-lunar server-todo verification-needed verification-needed-jammy verification-needed-lunar
2023-10-16 14:15:47 Arnaud Veron tags server-todo verification-needed verification-needed-jammy verification-needed-lunar server-todo verification-done-jammy verification-needed verification-needed-lunar
2023-10-24 15:57:35 Mitchell Dzurick tags server-todo verification-done-jammy verification-needed verification-needed-lunar server-todo verification-done verification-done-jammy verification-done-lunar
2023-11-07 14:22:31 Launchpad Janitor multipath-tools (Ubuntu Lunar): status Fix Committed Fix Released
2023-11-07 14:22:40 Timo Aaltonen removed subscriber Ubuntu Stable Release Updates Team
2023-11-07 14:25:58 Launchpad Janitor multipath-tools (Ubuntu Jammy): status Fix Committed Fix Released