Activity log for bug #1960537

Date Who What changed Old value New value Message
2022-02-10 18:07:53 John Chittum bug added bug
2022-02-10 21:55:28 Brian Murray nominated for series Ubuntu Jammy
2022-02-10 21:55:28 Brian Murray bug task added livecd-rootfs (Ubuntu Jammy)
2022-02-10 21:55:45 Brian Murray livecd-rootfs (Ubuntu Jammy): assignee John Chittum (jchittum)
2022-02-10 21:55:50 Brian Murray livecd-rootfs (Ubuntu Jammy): status New In Progress
2022-02-10 21:55:55 Brian Murray livecd-rootfs (Ubuntu Jammy): importance Undecided High
2022-02-11 01:35:24 Steve Langasek livecd-rootfs (Ubuntu Jammy): status In Progress Fix Committed
2022-02-11 01:35:28 Steve Langasek livecd-rootfs (Ubuntu Impish): status New Fix Committed
2022-02-11 01:35:30 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2022-02-11 01:35:32 Steve Langasek bug added subscriber SRU Verification
2022-02-11 01:35:35 Steve Langasek tags verification-needed verification-needed-impish
2022-02-11 01:37:31 Steve Langasek description When running builds locally or within launchpad, cloud-images are starting to fail in many places with the following error: ``` + udevadm settle + '[' -n /dev/mapper/loop0p1 -a -b /dev/mapper/loop0p1 ']' + '[' -e /etc/mtab ']' + e2fsck -y -E discard /dev/mapper/loop0p1 e2fsck 1.45.5 (07-Jan-2020) /dev/mapper/loop0p1 is in use. e2fsck: Cannot continue, aborting. ``` This call happens on calls to `umount_disk_image` which then calls `umount_partition`. `dmsetup info` reports (as an example) ``` Name: loop3p1 State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 0 Number of targets: 1 UUID: part1-devnode_7:3_Wh5pYvM ``` This shows that there's an open file handle on loop3p1. This was first noticed in Jammy builds. After a fair amount of debugging by jchittum, no root cause was found. However, a workaround was added to the failing CPC hook: sleep 30 Trial and error found that shorter sleeps still failed. Due to the ephemeral nature, I [jchittum] have been unable to find a root cause. The file handle cleans itself up quickly enough that even when I'm on a node watching the build, I haven't been able to track down an exact cause due to the handle releasing in a short time. When running builds locally or within launchpad, cloud-images are starting to fail in many places with the following error: ``` + udevadm settle + '[' -n /dev/mapper/loop0p1 -a -b /dev/mapper/loop0p1 ']' + '[' -e /etc/mtab ']' + e2fsck -y -E discard /dev/mapper/loop0p1 e2fsck 1.45.5 (07-Jan-2020) /dev/mapper/loop0p1 is in use. e2fsck: Cannot continue, aborting. ``` This call happens on calls to `umount_disk_image` which then calls `umount_partition`. `dmsetup info` reports (as an example) ``` Name: loop3p1 State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 0 Number of targets: 1 UUID: part1-devnode_7:3_Wh5pYvM ``` This shows that there's an open file handle on loop3p1. This was first noticed in Jammy builds. After a fair amount of debugging by jchittum, no root cause was found. However, a workaround was added to the failing CPC hook: sleep 30 Trial and error found that shorter sleeps still failed. Due to the ephemeral nature, I [jchittum] have been unable to find a root cause. The file handle cleans itself up quickly enough that even when I'm on a node watching the build, I haven't been able to track down an exact cause due to the handle releasing in a short time. [Test case] The autopkgtests should continue to pass. For focal, the daily image builds should continue to pass. For all series, the CPC team will need to confirm that this has not regressed their builds, as they downstream from the livecd-rootfs in the archive using cloud sauce scripts.
2022-02-11 01:41:37 Steve Langasek livecd-rootfs (Ubuntu Focal): status New Fix Committed
2022-02-11 01:41:42 Steve Langasek tags verification-needed verification-needed-impish verification-needed verification-needed-focal verification-needed-impish
2022-02-11 01:42:17 Steve Langasek livecd-rootfs (Ubuntu Bionic): status New Fix Committed
2022-02-11 01:42:22 Steve Langasek tags verification-needed verification-needed-focal verification-needed-impish verification-needed verification-needed-bionic verification-needed-focal verification-needed-impish
2022-02-11 03:17:11 Launchpad Janitor livecd-rootfs (Ubuntu Jammy): status Fix Committed Fix Released
2022-02-11 16:46:00 John Chittum tags verification-needed verification-needed-bionic verification-needed-focal verification-needed-impish verification-done-bionic verification-done-focal verification-done-impish verification-needed
2022-02-11 17:11:28 Brian Murray tags verification-done-bionic verification-done-focal verification-done-impish verification-needed verification-done verification-done-bionic verification-done-focal verification-done-impish
2022-02-11 21:10:04 Steve Langasek removed subscriber Ubuntu Stable Release Updates Team
2022-02-11 21:10:25 Launchpad Janitor livecd-rootfs (Ubuntu Focal): status Fix Committed Fix Released
2022-02-11 21:10:51 Launchpad Janitor livecd-rootfs (Ubuntu Bionic): status Fix Committed Fix Released
2022-02-15 18:53:00 Launchpad Janitor livecd-rootfs (Ubuntu Impish): status Fix Committed Fix Released