multipathd.service not restarted after upgrade

[Mitchell Dzurick]

[ Important info regarding this bug ]
This bug is blocking the phasing of the [lunar|jammy]-proposed multipath-tools package, so it is important that when uploaded, the changes from the proposed pocket are included.

While testing the fix for this issue, a new bug was noticed, that on certain operations (upgrade/fresh install) of multipath-tools, the multipathd.socket would be in an inactive state. This is not addressed in this bug, because it is present in Mantic and a proper solution is being found for that and will potentially be SRU'd later.

New bug found - https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2034471

[ Impact ]

Impacted versions are in proposed

(lunar-proposed) 0.8.8-1ubuntu2.1
(jammy-proposed) 0.8.8-1ubuntu1.22.04.2

In the package currently in proposed (see versions above), this is the sequence of events during a package upgrade:

1) "old-prerm upgrade <vers>":
- "invoke-rc.d multipath-tools stop" is only called if "$1" is "remove", which is not the case here.
- "deb-systemd-invoke stop multipathd.socket" does not stop the service, just the socket
Therefore, at the end of this script, we have the socket stopped, but the service running

2) "old-postrm upgrade <vers>":
- all service actions are conditioned on "$1" being "remove" or "purge", not the case here

3) "new-postinst configure <vers>"
- here we reach the point where "$1" is checked for "configure", and then deb-systemd-invoke start is called on the service and the socket. Except, since the service is still running, and wasn't stopped by prerm, starting it is a noop. Boom, package upgrade is finished and service was not restarted.

[ Fix ]

The previous multipath-tools.prerm maintainer script did not stop the multipathd.service unit, therefore it needs to be stopped in postinst for this specific upgrade case.

"systemctl daemon-reload" is specifically not being called before deb-systemd-invoke because:
  a) it will be called later in the script, before multipathd.service is started, so let's not call it twice
  b) it has system-wide effects, like running all systemd generators again
  c) we don't really need the new multipathd.service unit file to stop the old running daemon. In fact, one could argue it's more correct to use the old unit file to stop the old daemon.

This approach will generate a one-time warning:
```
Warning: The unit file, source configuration file or drop-ins of multipathd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
```

[ Test Plan ]

The current dep8 test is sufficient for testing functionality of multipath-tools, so the test plan for this update is targetting the upgrade behavior and additional scrutiny on multipath-tools-boot. multipath-tools-boot is explicitly included in this testing because it is not covered in dep8 and regressions in this package can cause systems to not boot, which makes it important to test properly.

Tests are written for lunar, but will be the similar for Jammy. Just change the lxc image and the version numbers

1) On upgrade
$ lxc launch ubuntu:lunar l --vm
$ lxc shell l
# dpkg -l multipath-tools | grep multipath-tools | awk '{print $3}'
0.8.8-1ubuntu2
# pidof multipathd
<some number>
# apt install -y multipath-tools=0.8.8-1ubuntu2.2
# pidof multipathd
<should be a different number than before>
# systemctl status multipathd.service multipathd.socket | grep Active -B2
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-09-20 12:40:09 UTC; 17s ago
--
● multipathd.socket - multipathd control socket
     Loaded: loaded (/lib/systemd/system/multipathd.socket; enabled; preset: enabled)
     Active: active (running) since Wed 2023-09-20 12:40:09 UTC; 17s ago

2) Installing a fresh package
$ lxc launch ubuntu:lunar l --vm
$ lxc shell l
# apt purge -y multipath-tools
# apt update -y
# apt install -y multipath-tools=0.8.8-1ubuntu2.2
# systemctl status multipathd.service multipathd.socket | grep Active -B2
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-09-20 12:41:27 UTC; 3s ago
--
○ multipathd.socket - multipathd control socket
     Loaded: loaded (/lib/systemd/system/multipathd.socket; enabled; preset: enabled)
     Active: inactive (dead)

3) Upgrading to future packages
This requires making a test package > 0.8.8-1ubuntu2.2

$ lxc launch ubuntu:lunar l --vm
$ lxc shell l
# dpkg -l multipath-tools | grep multipath-tools | awk '{print $3}'
0.8.8-1ubuntu2.2
# pidof multipathd
<some number>
# apt install -y multipath-tools=0.8.8-1ubuntu2.3
# pidof multipathd3
<should be a different number then before>
# systemctl status multipathd.service multipathd.socket | grep Active -B2
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-09-20 12:42:01 UTC; 3s ago
--
○ multipathd.socket - multipathd control socket
     Loaded: loaded (/lib/systemd/system/multipathd.socket; enabled; preset: enabled)
     Active: inactive (dead)

4)
Also tested is multipath-tools-boot manually as a sanity check that booting still occurs. This is done manually like so:

$ download latest lunar-live-server-amd64.iso
$ fallocate -l 20G image.img
$ kvm -m 2048 -boot d -cdrom ./lunar-live-server-amd64.iso -device virtio-scsi-pci,id=scsi -drive file=image.img,if=none,id=sda,format=raw,file.locking=off -device scsi-hd,drive=sda,serial=0001 -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off -device scsi-hd,drive=sdb,serial=0001
Install like regular, reboot. Stop reboot process.
$ kvm -m 2048 -boot c -cdrom ./lunar-live-server-amd64.iso -device virtio-scsi-pci,id=scsi -drive file=image.img,if=none,id=sda,format=raw,file.locking=off -device scsi-hd,drive=sda,serial=0001 -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off -device scsi-hd,drive=sdb,serial=0001

After changing to -boot c, ensure you can login and check that the root partition is using the multipathd devices with lsblk and multipath -l

[ Where Problems could Occur ]
* Users could mistake the newly generated one-time warning as an error
* Upgrading will make the socket inactive (this is mentioned above as a known bug, so it is not a regression due to this fix)

[Timo Aaltonen (tjaalton)]

Hello Mitchell, or anyone else affected,

Accepted multipath-tools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.8-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Mitchell, or anyone else affected,

Accepted multipath-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.8.8-1ubuntu1.22.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mitchell Dzurick (mitchdz)]

Lunar is tested. There are multiple tests here, and I did some extra tests that are not in the testplan. Due to the amount of data that is tested, I attached it as a .txt file, but the following tests were performed:

1. Testing that multipathd.service is restarted on upgrade from the current package
2. Testing downgrade to previous package
3. Installing a fresh package (no warning message here because dpkg compare is lt-nl)
4. Upgrading to a future package
5. Testing functionality after upgrading to the new package with multipath devices
6. Testing that multipath devices are still usable after upgrading

The only contentious thing I noticed is that on upgrade or install, the socket is inactive. This is captured already in https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2034471 but worth mentioning here. The utility works fine even with this.

[Mitchell Dzurick (mitchdz)]

Jammy is tested. There are multiple tests here, and I did some extra tests that are not in the testplan. Due to the amount of data that is tested, I attached it as a .txt file, but the following tests were performed:

1. Testing that multipathd.service is restarted on upgrade from the current package
2. Testing downgrade to previous package
3. Installing a fresh package (no warning message here because dpkg compare is lt-nl)
4. Upgrading to a future package
5. Testing functionality after upgrading to the new package with multipath devices
6. Testing that multipath devices are still usable after upgrading

The only contentious thing I noticed is that on upgrade or install, the socket is inactive. This is captured already in https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/2034471 but worth mentioning here. The utility works fine even with this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.8.8-1ubuntu2.2

---------------
multipath-tools (0.8.8-1ubuntu2.2) lunar; urgency=medium

  * debian/multipath-tools.postinst: restart multipathd when upgrading from lt
    0.8.8-1ubuntu2.2 (LP: #2035098)

multipath-tools (0.8.8-1ubuntu2.1) lunar; urgency=medium

  * d/p/0018-libmpathpersist-fix-command-keyword-ordering.patch:
    Fix mpathpersist command keyword ordering. (LP: #2026881)
  * Do not install init scripts (LP: #2000186)
    - d/multipath-tools.maintscript: remove /etc/init.d/multipath-tools
    - d/rules: do not call dh_installinit

 -- Mitchell Dzurick <email address hidden> Wed, 20 Sep 2023 05:18:54 -0700

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for multipath-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package multipath-tools - 0.8.8-1ubuntu1.22.04.3

---------------
multipath-tools (0.8.8-1ubuntu1.22.04.3) jammy; urgency=medium

  * debian/multipath-tools.postinst: restart multipathd when upgrading from lt
    0.8.8-1ubuntu1.22.04.3 (LP: #2035098)

multipath-tools (0.8.8-1ubuntu1.22.04.2) jammy; urgency=medium

  [ Sergio Durigan Junior ]
  * d/p/0018-libmpathpersist-fix-command-keyword-ordering.patch:
    Fix mpathpersist command keyword ordering. (LP: #2026881)

  [ Mitchell Dzurick ]
  * Do not install init scripts (LP: #2000186)
    - d/multipath-tools.maintscript: remove /etc/init.d/multipath-tools
    - d/rules: do not call dh_installinit

 -- Mitchell Dzurick <email address hidden> Wed, 20 Sep 2023 08:05:06 -0700