New multipath-tools doesn't update in containers (disco)

[Christian Ehrhardt ]

Steps to reproduce
1. use current Disco container in LXD
 $ lxc launch ubuntu-daily:d d-upgr
2. dist-upgrade to do the multipath-tools upgrade
 Note: dist upgrade will trigger the same, but this has less noise
 $ apt install multipath-tools

Right after start we have this.
root@d-upgr:~# apt-cache policy multipath-tools
multipath-tools:
  Installed: 0.7.4-2ubuntu4
  Candidate: 0.7.4-2ubuntu7
  Version table:
     0.7.4-2ubuntu7 500
        500 http://archive.ubuntu.com/ubuntu disco/main amd64 Packages
 *** 0.7.4-2ubuntu4 100
        100 /var/lib/dpkg/status

Example:
root@d-upgr:~# apt install multipath-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  kpartx
Suggested packages:
  multipath-tools-boot
The following packages will be upgraded:
  kpartx multipath-tools
2 upgraded, 0 newly installed, 0 to remove and 23 not upgraded.
Need to get 278 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu disco/main amd64 kpartx amd64 0.7.4-2ubuntu7 [27.8 kB]
Get:2 http://archive.ubuntu.com/ubuntu disco/main amd64 multipath-tools amd64 0.7.4-2ubuntu7 [250 kB]
Fetched 278 kB in 1s (434 kB/s)
(Reading database ... 28913 files and directories currently installed.)
Preparing to unpack .../kpartx_0.7.4-2ubuntu7_amd64.deb ...
Unpacking kpartx (0.7.4-2ubuntu7) over (0.7.4-2ubuntu4) ...
Preparing to unpack .../multipath-tools_0.7.4-2ubuntu7_amd64.deb ...
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Check that device-mapper is available in the kernel.
Incompatible libdevmapper 1.02.145 (2017-11-03) and kernel driver (unknown version).
Command failed
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.18.0-17-generic/modules.dep.bin'
modprobe: FATAL: Module dm-multipath not found in directory /lib/modules/4.18.0-17-generic
Unpacking multipath-tools (0.7.4-2ubuntu7) over (0.7.4-2ubuntu4) ...
Setting up kpartx (0.7.4-2ubuntu7) ...
Setting up multipath-tools (0.7.4-2ubuntu7) ...
multipathd.socket is a disabled or a static unit, not starting it.
Job for multipathd.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status multipathd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript multipath-tools, action "start" failed.
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Mon 2019-04-15 08:50:07 UTC; 6ms ago
  Process: 1708 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=1/FAILURE)
  Process: 1709 ExecStart=/sbin/multipathd -d -s (code=dumped, signal=ABRT)
 Main PID: 1709 (code=dumped, signal=ABRT)
   Status: "configure"

Apr 15 08:50:07 d-upgr multipathd[1709]: read /etc/multipath.conf
Apr 15 08:50:07 d-upgr multipathd[1709]: path checkers start up
Apr 15 08:50:07 d-upgr multipathd[1709]: /dev/mapper/control: open failed: Permission denied
Apr 15 08:50:07 d-upgr multipathd[1709]: Failure to communicate with kernel device-mapper driver.
Apr 15 08:50:07 d-upgr multipathd[1709]: Check that device-mapper is available in the kernel.
Apr 15 08:50:07 d-upgr multipathd[1709]: Incompatible libdevmapper 1.02.145 (2017-11-03) and kernel driver (unknown version).
Apr 15 08:50:07 d-upgr multipathd[1709]: (urcu-call-rcu-impl.h:call_rcu_data_init@438) Unrecoverable error: Resource temporarily unavailable
Apr 15 08:50:07 d-upgr systemd[1]: multipathd.service: Main process exited, code=dumped, status=6/ABRT
Apr 15 08:50:07 d-upgr systemd[1]: multipathd.service: Failed with result 'core-dump'.
Apr 15 08:50:07 d-upgr systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
dpkg: error processing package multipath-tools (--configure):
 installed multipath-tools package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (240-6ubuntu4) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.29-0ubuntu1) ...
Errors were encountered while processing:
 multipath-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

A quick check with former versions showed that this is no new behavior (e.g. in Bionic multipath-tools also fails to install).
The actual "new breaking change" seems to be that the LXD image now contains multipath.

It would be nice to fix multipath to behave better in a containerized environment, so a bug task for multipath certainly is correct.

But also the image building for LXD should not by default have that installed right?
Was there a reason to do so and if so has that been tested (as it seems no).
Was it a seed or dependency change?

[Christian Ehrhardt  (paelzer)]

Added a LXD task for the LXD images - but I think they only consume what is minimal image or so, I do not expect it to be fixed on LXD upstream or the package - this is more a FYI.

[Christian Ehrhardt  (paelzer)]

root@d-upgr:~# apt remove multipath-tools -s
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  ibverbs-providers kpartx libaio1 libfreetype6 libibverbs1 libnl-3-200 libnl-route-3-200 libnspr4 libnss3 librados2 libsgutils2-2 liburcu6 sg3-utils sg3-utils-udev
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  multipath-tools ubuntu-server
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.

Ok, it is seeded in ubuntu-server which I think is no news [1], it might formerly have been stripped ?

Checking the meta packages:
Disco:
$ apt-cache show ubuntu-server | grep '^Depends'
...
Yes multipath-tools is a new dependency in Disco, but not in older versions.
Interesting, checking ubuntu-meta and co

[1]: https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/commit/?id=8191ce8450b1312513df1cdcaf21039ef4606ee3

[Christian Ehrhardt  (paelzer)]

Change introduced in Ubuntu-meta in 1.431 [1]
But since that usually only reflects the seeds that must be from there somehow.

[1]: https://launchpad.net/ubuntu/+source/ubuntu-meta/1.431

[Christian Ehrhardt  (paelzer)]

Germinate [2] clearly points at the seeds.
  bin:multipath-tools src:multipath-tools why:Ubuntu.Disco server seed
  bin:multipath-tools-boot src:multipath-tools why:Ubuntu.Disco server-ship-live seed

The second I saw before in the seeds [1], but where exactly is the first line listed here within the seeds.

Interesting is, that this is listed similarly in cosmic [3] but the reason is then only the multipath-tools-boot.
  bin: multipath-tools src:multipath-tools why:multipath-tools-boot

That makes it appear in disco [4] but not in cosmic [5].
And the ubuntu-server in the ubuntu-meta package is derived from [4].

I need to once more check the seeds if there was any multipath'y change recently ...

[1]: https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/commit/?id=8191ce8450b1312513df1cdcaf21039ef4606ee3
[2]: http://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.disco/all+extra
[3]: http://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.cosmic/all+extra
[4]: http://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.disco/server
[5]: http://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.cosmic/server

[Christian Ehrhardt  (paelzer)]

Yep, once checking the right and up to date branches it seems clear.
I found [1], subscribing xnox here as well.

This change was on late February, there was no ubuntu-meta update since then so the issue wasn't triggered.

I think we'd want to undo the seed change, but in a fashion so that what xnox wanted to fix still works. We wanted it on live (for installers) and ship (for installers as well), but not *everywhere* for reasons like this.

@xnox - could you outline what the change was supposed to fix exactly so that I don't blindly revert it breaking your old case?

[1]: https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/commit/?id=9c1df562

[Christian Ehrhardt  (paelzer)]

Now that we know that we can drop all bug ubuntu-meta task, marking the others invalid.

[Christian Ehrhardt  (paelzer)]

This was initially triggered by:
https://code.launchpad.net/~xnox/ubuntu-seeds/+git/ubuntu-seeds/+merge/363541

Which was matching cleanups we did earlier this year and again in Malta.

But that there was fallout to multipath-tools was unknown at least to me, thanks @xnox for the dup to the bug about that.