buildd daily kinetic, lunar, and mantic image missing systemd-resolved

[Callahan Kovacs]

[ Impact ]

 * There was a regression in the buildd images when systemd-resolved was moved
   out of systemd to a separate package, as the buildd recipes did not include
   systemd-resolved. Those images don't include a resolvconf anymore.

 * This affects buildd images only in the context where they are booted
   standalone (e.g. kvm, multipass). buildd containers which get network
   configuration injected (like launchpad) are not affected.

 * This fix adds the systemd-resolved package explicitly, making the resolver
   available on buildd images like pre-kinetic.

[ Test Plan ]

 * Build image with project=ubuntu-base subproject=buildd image_target=all

 * Boot qcow2 buildd image. run `apt-get update`

 * Apt should be able to resolve mirrors and update fine.

[ Where problems could occur ]

 * If the context in which buildd images are used injects a resolv.conf
   without unlinking the systemd-resolved stub, the resolver might revert
   the injected config to the systemd-resolved stub on restart.

[ Original Description ]

Hello,

I'm trying to use the buildd daily images [1]. However, I'm having an issue with `systemd-resolved` not being installed on buildd's daily kinetic and lunar images.

From what I can gather, systemd-resolved:
1. exists in all other buildd daily images and buildd release images
2. exists in the other cloud images [2]
3. was separated from the systemd package in kinetic and lunar [3]

It's a bit of a catch-22, because without `systemd-resolved` or literally `/usr/bin/resolvectl`, I can't run `apt install systemd-resolved` in my LXD environment.

Can someone please advise? Was this package excluded on purpose?

Sources:
[1] https://cloud-images.ubuntu.com/buildd/daily/
[2] https://cloud-images.ubuntu.com/
[3] https://packages.ubuntu.com/search?keywords=systemd-resolved&searchon=names&exact=1&suite=all&section=all

[John Chittum (jchittum)]

starting in kinetic, systemd-resolved was separated into its own package. as i look in the livecd-rootfs code for buildd, i'm a little confused as to its package set -- i don't see an identified seed, just a copy from the existing chroot and the addition of a couple packages. which is awkward because the initial chroot is _also_ builds.

as i keep digging through, nothing is installing `systemd-resolved`, which makes me think it's missing in the initial package definitions of buildd. it's not listed as an a package to install during the config step:

https://git.launchpad.net/livecd-rootfs/tree/live-build/auto/config#n1006

this isn't affecting launchpad-buildd because of their specific networking setup. since systemd packaged resolved with it before kinetic, it feels safe to me to add in the split out package to the list in live-build/auto/config

I'm checking with cjwatson on that assertion though, as we don't want to introduce issues for launchpad.

[Callahan Kovacs (mr-cal)]

Thanks for looking into it further and the quick response!

[Sergio Schvezov (sergiusens)]

Any updates on this?

[John Chittum (jchittum)]

not yet. making changes to buildd can be a bit of dangerous business. it could affect the launchpad build system, which means breaking all builds. it should be safe, but i'm not even sure how we can test the assertion right now (and Colin is very busy with another $BIG_THING)

internally, i've proposed a group of us getting together again about the best path forward for snapcraft. i personally have some worries about `buildd` being used outside of launchpad builds, since it was originally very specifically designed for that purpose. i know we talked about big package diffs between minimal and buildd, but also buildd has a lot of very specific configuration that, if changed, could break launchpad.

[John Chittum (jchittum)]

that being said, i've added `livecd-rootfs`, as that is where any of the buildd changes must occur. it is an open source package, so anyone can open an MP. I also can't drive the merge and package release of `livecd-rootfs`. we'll need to work with a core-dev for that part.

[Steve Langasek (vorlon)]

I do think this is an oversight that systemd-resolved is not included in the buildd image after the split, that just happened to work in Launchpad. I am +1 for getting the package added back via livecd-rootfs.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Callahan Kovacs (mr-cal)]

Hey @jchittum, any update or timeline on this? Thanks

[John Chittum (jchittum)]

Sorry, really busy summer. We've prioritized this fix for the next pulse, so hopefully we'll have this in and tested in the next 3-4 weeks.

[Callahan Kovacs (mr-cal)]

@simpoir and @jchittum, do you have any updates?

The core24 base snap is ready to be built on noble and soon we'll be supporting devel builds using core24. We really need to be able to use the buildd base.

Thanks!

[Sergio Schvezov (sergiusens)]

This is blocking progress on Ubuntu Core 24

[Steve Langasek (vorlon)]

lunar is EOL in less than 2 months and has had this problem the entire time, it's not worth fixing there now; but we should still fix mantic which is the current stable release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 24.04.7

---------------
livecd-rootfs (24.04.7) noble; urgency=medium

  [ Dave Jones ]
  * Remove more bits related to pre-installed desktop images (LP: #2038098)

  [ Jean-Baptiste Lallement ]
  * Include -updates when calling germinate. LP: #1921862.

  [ Dan Bungert ]
  * cpc/vagrant: fix possible autopkgtest hang on ssh-keygen

  [ Simon Poirier ]
  * add systemd-resolved back to buildd images. LP: #2007419.

  [ Steve Langasek ]
  * live-build/functions: add additional debugging to mount_image() on
    failure since we again have loop partitions failing to be block devices.

 -- Steve Langasek <email address hidden> Sun, 03 Dec 2023 21:38:27 -0800

[Timo Aaltonen (tjaalton)]

Hello Callahan, or anyone else affected,

Accepted livecd-rootfs into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/23.10.59 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.