Installing/Upgrading multipath-tools leads to multipathd.socket being inactive

[Mitchell Dzurick]

This is an issue with the systemd service not stating the dependency on the socket, but stating a start order with the socket. The fix here is to add

Wants=multipathd.socket

in multipathd.service which is a weak dependency, so the chance of issues arising from it is low, but helps to solve this particular issue.

[Impact]

On Upgrade/Install, the multipathd.socket states that it is inactive. An error is seen during install/upgrade that says

Job failed. See "journalctl -xe" for details.

This issue affects the following packages:

Lunar (0.8.8-1ubuntu2.1)
Jammy (0.8.8-1ubuntu1.22.04.2)
Focal (0.8.3-1ubuntu2.2)
- On upgrade
- On first install

Mantic (0.9.4-5ubuntu3)
- On first install

The issue is because of the order of the systemctl command and the systemd unit files.

multipathd.service has:
...
After=multipathd.socket
...
Also=multipathd.socket

multipath-tools.postinst has:
Jammy/Lunar/Focal (added by dh_systemd_start)
deb-systemd-invoke start 'multipathd.service' 'multipathd.socket' >/dev/null || true

Mantic (added by dh_installsystemd)
if [ -n "$2" ]; then
    _dh_action=restart
else
    _dh_action=start
fi
deb-systemd-invoke $_dh_action 'multipathd.service' 'multipathd.socket' >/dev/null || true

In other words,
systemctl stop multipathd.service multipathd.socket
systemctl start multipathd.service multipathd.socket

will fail

systemctl stop multipathd.service multipathd.socket
systemctl start multipathd.socket multipathd.service

will succeed

If we try to start the socket after the service, we get a failure. This is by design so multiple processes of the socket are not active at once.

[Test Cases]
Due to the nature of this bug/fix, multiple test cases are defined here:
1. Install multipath-tools when it is uninstalled
2. Upgrade multipath-tools to this version
3. Downgrade multipath-tools to this version
4. Reinstall multipath-tools when it is this version
5.
systemctl stop multipathd.service multipathd.socket
systemctl start multipathd.service multipathd.socket

All test cases should result in _no errors_ and both multipathd.service multipathd.socket are active

[Reproducer]
$ lxc launch ubuntu-daily:mantic m --vm
$ lxc shell m
# apt purge -y multipath-tools
# apt update -y
# apt install -y multipath-tools
# systemctl status multipathd.service multipathd.socket | grep Active -B2
● multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-09-13 19:30:56 UTC; 47s ago
--
○ multipathd.socket - multipathd control socket
     Loaded: loaded (/lib/systemd/system/multipathd.socket; enabled; preset: enabled)
     Active: inactive (dead)

[Mitchell Dzurick (mitchdz)]

This is because postinst attempts to start multipathd.socket while multipathd.service is active.

It's an issue with ordering of stopping/starting the service/socket.

Attempting to start the socket while the service is active will result in a failure due to how the socket will attempt to activate the service. To properly stop/start this socket/service the ordering is important, and we must do:

# systemctl stop multipathd.service multipathd.socket
# systemctl start multipathd.socket multipathd.service

Notice how the service is stopped first, but started last.

Inside of d/rules we have

dh_systemd_start -pmultipath-tools multipathd.service

Which results in the following line of postinst:
# Automatically added by dh_systemd_start/13.11.4ubuntu3
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
        if [ -z "${DPKG_ROOT:-}" ] && [ -d /run/systemd/system ]; then
                systemctl --system daemon-reload >/dev/null || true
                deb-systemd-invoke start 'multipathd.service' 'multipathd.socket' >/dev/null || true
        fi
fi
# End automatically added section

Which postinst should start the socket first.

[Mitchell Dzurick (mitchdz)]

This is not specific to reinstalling the package, it will show up on upgrade as well.

[Mitchell Dzurick (mitchdz)]

Adding a dependency requirement in multipathd.service fixes this issue. This is similar to what is done with uuidd.service.

uuidd.service (as an example) has
Required=uuidd.socket

Which enforces a (hard) dependency requirement that systemd works with. Required= makes it so that if the socket does not start, the service will also not start[0].

I would suggest adding a weaker dependency[1] using Want= in the service file. This satisfies fixing the issue in this bug and has a lower chance of causing other issues.

[0] - https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=

[1] - https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Wants=

[Mitchell Dzurick (mitchdz)]

Attempted to upstream[0] the fix of adding Wants=multipathd.socket in multipathd.service and was rejected, as upstream wants the ability to start the service without the socket starting.

[0] - https://github.com/opensvc/multipath-tools/pull/72

[Mitchell Dzurick (mitchdz)]

Also affects Noble.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mitchell Dzurick (mitchdz)]

Putting this bug on hold until upstream finalizes what they are doing for socket based activation. Related bug - https://github.com/opensvc/multipath-tools/pull/78

I will update the description and make new MPs once this is done.

[Brian Murray (brian-murray)]

Ubuntu 23.04 (Lunar Lobster) has reached end of life, so this bug will not be fixed for that specific release.