livecd-rootfs focal hangs during kpartx (ubuntu-cpc project)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
launchpad-buildd |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Focal livecd-rootfs builds of the ubuntu-cpc project hang indefinitely when 'kpartx -av binary/
In the build container the kpartx process is in the sleeping state.
The container journalctl output has messages like:
systemd-
The host journalctl output has messages like:
kernel: blk_update_request: I/O error, dev loop0, sector 0
I'll attach the build output up to the hang as https:/
The lxd environment is a privileged container and udev won't create /dev/dm-# devices in /dev because devtmpfs is used in the host but tmpfs is used in the container. There is some change in behavior after Oct 30th in Focal that exposes this failure. In our image manifests we see the following changes between the working and failing build:
(MAN is the version of the package in the manifest for the working build while ARCHIVE is the version in the archive on the day of the first failing build)
bzip2:amd64 (MAN: 1.0.6-9.2) (ARCHIVE: 1.0.8-2)
gcc-9-base:amd64 (MAN: 9.2.1-12ubuntu1) (ARCHIVE: 9.2.1-15ubuntu1)
libbz2-1.0:amd64 (MAN: 1.0.6-9.2) (ARCHIVE: 1.0.8-2)
libgcc1:amd64 (MAN: 1:9.2.1-12ubuntu1) (ARCHIVE: 1:9.2.1-15ubuntu1)
libnss-
libpam-
libsoup2.4-1:amd64 (MAN: 2.68.2-0ubuntu1) (ARCHIVE: 2.68.2-1)
libstdc++6:amd64 (MAN: 9.2.1-12ubuntu1) (ARCHIVE: 9.2.1-15ubuntu1)
libsystemd0:amd64 (MAN: 242-7ubuntu3) (ARCHIVE: 243-2ubuntu1)
libudev1:amd64 (MAN: 242-7ubuntu3) (ARCHIVE: 243-2ubuntu1)
netcat-
python3-
systemd:amd64 (MAN: 242-7ubuntu3) (ARCHIVE: 243-2ubuntu1)
systemd-sysv:amd64 (MAN: 242-7ubuntu3) (ARCHIVE: 243-2ubuntu1)
ubuntu-
udev:amd64 (MAN: 242-7ubuntu3) (ARCHIVE: 243-2ubuntu1)
I tried back-leveling just udev:
# dpkg -i udev_242-
(Reading database ... 20790 files and directories currently installed.)
Preparing to unpack udev_242-
Unpacking udev (242-7ubuntu3) over (242-7ubuntu3) ...
dpkg: warning: downgrading libudev1:amd64 from 243-3ubuntu1 to 242-7ubuntu3
Preparing to unpack libudev1_
Unpacking libudev1:amd64 (242-7ubuntu3) over (243-3ubuntu1) ...
Setting up libudev1:amd64 (242-7ubuntu3) ...
Setting up udev (242-7ubuntu3) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for systemd (243-3ubuntu1) ...
Processing triggers for libc-bin (2.30-0ubuntu2) ...
Processing triggers for initramfs-tools (0.133ubuntu11) ...
# systemctl restart systemd-
The problem still recreated with kpartx hanging in the build with this change but maybe still be close. However, I am able to address the kpartx hang by making the /dev/dm-X nodes in launchpad-buildd or changing the livecd-rootfs hooks to call 'kpartx -anv binary/
On that kpartx invocation change, it doesn't appear that kpartx has changed in focal so a behavior it depends on appears to have changes. Also I'd like to avoid changing the pattern for calling kpartx due to use in various locations of livecd-rootfs.
So I'm going to propose a patch to launchpad-buildd to add the /dev/dm-X nodes so we can have focal builds again, but we also should dig into the underlying cause of kpartx waiting indefinitely as that isn't excellent behavior.
Related branches
- Colin Watson (community): Approve
-
Diff: 48 lines (+19/-0)3 files modifieddebian/changelog (+6/-0)
lpbuildd/target/lxd.py (+7/-0)
lpbuildd/target/tests/test_lxd.py (+6/-0)
- Colin Watson (community): Approve
-
Diff: 221 lines (+85/-22)4 files modifieddebian/changelog (+7/-0)
debian/control (+1/-0)
lpbuildd/target/lxd.py (+19/-1)
lpbuildd/target/tests/test_lxd.py (+58/-21)
- Robert C Jennings: Pending requested
- Ubuntu Core Development Team: Pending requested
-
Diff: 39 lines (+15/-1)2 files modifieddebian/changelog (+7/-0)
live-build/functions (+8/-1)
tags: | added: id-5dc15a912d7f90886388bfd5 |
I managed to reproduce this in a xenial vm on my machine. The problem appears with udev 243 it seems (installing udev and libudev 242-7ubuntu3 makes it work again).
When kpartx works it prints this message:
root@c:~# kpartx -av image.img
/dev/mapper/loop0p1 not set up by udev: Falling back to direct node creation.
add map loop0p1 (252:0): 0 2088960 linear 7:0 4096
so for some reason it's not getting to that fallback. FWIW, that fallback code is in libdevmapper (src:lvm2).