Ubuntu Desktop Canary ISO failing to build since builders update

[Sebastien Bacher]

the desktop canary image started failing a week ago (https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/jammy/ubuntu-canary)

/snap/gtk-common-themes/1519: failed to setup loop device for /var/lib/snapd/snaps/gtk-common-themes_1519.snap.

it seems like maybe the builder is out of loop devices? the issue started the day the launchpad builders got upgraded to focal, unsure if the configuration changed?

[Sebastien Bacher (seb128)]

https://launchpad.net/ubuntu/+source/ubuntu-drivers-common/1:0.9.5 is a recent ubuntu build failure that hits a limit of open files on the builder which wasn't a problem before, could it be another case due to the upgrade?

[Sebastien Bacher (seb128)]

I've added some debugging to the ISO build, https://launchpadlibrarian.net/589592686/buildlog_ubuntu_jammy_amd64_ubuntu-canary_BUILDING.txt.gz

the journalctl log doesn't really have an error but the snapd team hinted that the system might be out of loop devices and a ls /dev shows only 8 of those on the builder which seems low, could that be a buildd configuration regression?

[Colin Watson (cjwatson)]

This may relate to either a newer version of lxd or a newer kernel, I'm not quite sure; but either way, https://discuss.linuxcontainers.org/t/udev-inside-privileged-container/13314 indicates that in the sort of privileged container setup we're using there's basically nothing for it but to create all the loop devices manually. I'm running a test to make sure that that works before committing to that approach.

[Paweł Stołowski (stolowski)]

I'm not sure this helps, but long time ago I introduced a snapd spread test to check that snap-preseed works inside lxc, mimicking lp builders at that time. Back then I found out that test would only work on 18.04 with lxd as a deb (which in fact reflected lp setup at that time), because lxd from snap would not allow `mount` inside the container (could be an issue with loop device creation as you suggested) - I poked manually with it and as far as I remember I was getting the same error you're reporting now.

Related remark in the test code: https://github.com/snapcore/snapd/blob/dd799e076969dd0be311232a130eb5d5dcf51a49/tests/main/preseed-lxd/task.yaml#L8

[Colin Watson (cjwatson)]

Fixed in launchpad-buildd 210.