live-build manipulation of conffiles results in them being treated as modified by end user

[Dan Bungert]

[ Impact ]

For packages that use preferences.d/ files, live-build will remove those files
causing the package to prompt for conffiles that it should have by default.

This issue was first seen in ubuntu-advantage-tools, which intended to use
conffiles in the 29 -> 29.2 transition but was prevented because of this bug.
The current plan is to have the conffiles be added in 29.5, but since ubuntu-
advantage-tools has an SRU exception this bug would prevent users on older than
mantic systems being able to run 29.5+ ubuntu-advantage-tools as intended.

To avoid removing packaged preferences.d files, the proposed patch only removes
/etc/apt/preferences.d/* files after we backup any existing preferences.d/
files in lb_chroot_archives install stage. Then during remove stage or
lb_chroot_archives the backed up packaged preferences.d files are restored.

[ Test Plan ]

Since there are no known packages that rely on conffiles in this way, the
easiest way to see the bug/test the patch is to use the test patch attached to
this bug report. This adds an additional step lb_chroot_create_preferences to
the build process that writes a preferences file /etc/apt/preferences.d/99dummy.
Then verify that /etc/apt/preferences.d/99dummy is present on a image built with
live-build; it will not be there if the bug is still present.

Individual steps to perform the test:

# Pull unpatched live-build
$ pull-lp-source live-build focal
$ cd live-build-3.0~a57/
# Add testing patch to live-build so behaviour is apparent
$ quilt import lb_chroot_archives-preferences_test.patch
$ quilt push

# Create an image using unpatched live-build
$ export UBUNTU_OLD_FASHIONED_DIR=[directory of ubuntu-bartender]
$ export LIVECD_ROOTFS_DIR=[directory of livecd-rootfs]
$ export LIVECD_ROOTFS_LOCAL_DIR=[directory of livecd-rootfs]
$ export LIVE_BUILD_DIR=[directory of unpatched live-build]
$ ubuntu-bartender --build-provider aws --livecd-rootfs-branch ubuntu/focalmaster -- --series focal --project ubuntu-cpc --image-target disk-image --subproject minimized

# Validate the dummy preferences file was created
$ grep 'Creating dummy preferences' [UNPATCHED]-ubuntu-bartender.log
+ Echo_message Creating dummy preferences.d files...
+ STRING=Creating dummy preferences.d files...
P:+ printf Creating dummy preferences.d files...\n
 Creating dummy preferences.d files...

# Download and extract the patched live-build
# Add testing patch to live-build so behaviour is apparent
$ quilt import lb_chroot_archives-preferences_test.patch
$ quilt push

# Create an image using patched live-build
$ export UBUNTU_OLD_FASHIONED_DIR=[directory of ubuntu-bartender]
$ export LIVECD_ROOTFS_DIR=[directory of livecd-rootfs]
$ export LIVECD_ROOTFS_LOCAL_DIR=[directory of livecd-rootfs]
$ export LIVE_BUILD_DIR=[directory of patched live-build]
$ ubuntu-bartender --build-provider aws --livecd-rootfs-branch ubuntu/focalmaster -- --series focal --project ubuntu-cpc --image-target disk-image --subproject minimized

# Validate the dummy preferences file was created
$ grep 'Creating dummy preferences' [PATCHED]-ubuntu-bartender.log
+ Echo_message Creating dummy preferences.d files...
+ STRING=Creating dummy preferences.d files...
P:+ printf Creating dummy preferences.d files...\n
 Creating dummy preferences.d files...

 # Mount the two images
 $ tar xvzf [UNPATCHED]-ubuntu-on-the-rocks.tar.gz
 $ mv build.output build.output-unpatched
 $ tar xvzf [PATCHED]-ubuntu-on-the-rocks.tar.gz
 $ mv build.output build.output-patched
 $ mkdir --parents --verbose ./mount-unpatched && sudo mount ./build.output-unpatched/livecd.ubuntu-cpc.ext4 ./mount-unpatched
 $ mkdir --parents --verbose ./mount-patched && sudo mount ./build.output-patched/livecd.ubuntu-cpc.ext4 ./mount-patched

 # Diff manifests (should be identical)
 $ diff build.output-patched/livecd.ubuntu-cpc.manifest build.output-unpatched/livecd.ubuntu-cpc.manifest

 # Create and diff filelists (patched version will have /etc/apt/preferences.d/99dummy; unpatched will not)
 $ (cd "./mount-unpatched" && sudo find -xdev) > "unpatched.filelist"
 $ sort unpatched.filelist > unpatched.filelist.sorted
 $ (cd "./mount-patched" && sudo find -xdev) > "patched.filelist"
 $ sort patched.filelist > patched.filelist.sorted
 $ diff patched.filelist.sorted unpatched.filelist.sorted
 230d229
< ./etc/apt/preferences.d/99dummy

# Clean up
$ sudo umount ./mount-unpatched
$ sudo umount ./mount-patched
$ rm -r *-unpatched
$ rm -r *-patched
$ rm *.filelist
$ rm *.filelist.sorted

[ Where problems could occur ]

Since we were not able to fully determine the history of lb_chroot_archives,
it's possible that there was a compelling reason for wiping out
/etc/apt/prferences.d/ in its entirety (the existing comment: probably too bold,
needs refinment (FIXME) suggest that perhaps this was not the long term plan but
there may have been a good reason nonetheless). Because of this uncertainty the
patch attempts to minimize the changes to lb_chroot_archives' existing function
but the backup and restore do keep files that would previously have been
deleted.

It would be preferable for packaged apt preference files to be stored somewhere
besides /etc but unfortunately this is how apt is written.

Since this creates copies of existing files temporarily there may be some
failure path where those copies are created but not properly cleaned up. In
that case there would be unexpected config files on the system in
/etc/apt/preferences.d.save. While untidy and not ideal, this state should not
create functional problems for the user as apt will not attempt to read the
copied files.

If anything in lb_chroot_archives creates different files in
/etc/apt/preferences.d with the same names as the saved versions, the saved
version of the file will overwrite the new file. We did not see this behavior
in testing but it could theoretically cause problems if it did occur. The
secion # Check local pinning preferences in lb_chroot_archives does write to the
directory explicitly.

[ Other Info ]

The original bug report, focusing on the ubuntu-advantage-tools issue has been
retained below for reference.

--------------------------------------------------------------------------------

When updating ubuntu-advantage-tools from 29 -> 29.2, a conffile prompt is seen.
This is on a system that is presumed to not have modified the relevant files.

Setting up ubuntu-advantage-tools (29.2) ...

Configuration file '/etc/apt/preferences.d/ubuntu-pro-esm-apps'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
 The default action is to keep your current version.
*** ubuntu-pro-esm-apps (Y/I/N/O/D/Z) [default=N] ?

Configuration file '/etc/apt/preferences.d/ubuntu-pro-esm-infra'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
 The default action is to keep your current version.
*** ubuntu-pro-esm-infra (Y/I/N/O/D/Z) [default=N] ?

steps to reproduce (at least today):
lxc launch ubuntu-daily:mantic
in container:
 apt update
 apt install ubuntu-advantage-tools

Discovered in subiquity CI - https://github.com/canonical/subiquity/actions/runs/6003040269/job/16280611441?pr=1773#step:3:1080

[John Chittum (jchittum)]

This looks to be a bug in `livecd-rootfs`. there's code from `live-build` that "helpfully" attempts to clean up. and in livecd-rootfs, we attempt to save off and restore.

https://git.launchpad.net/livecd-rootfs/tree/live-build/auto/build#n360

https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/chroot/100-preserve-apt-prefs.chroot

it appears something may be going awry here, because a launch of a current mantic image there is nothing in `/etc/apt/preferences.d/` when there should be a couple files dropped in by ubuntu-advantage-tools. this is verifiable by launching an lxc container of the current mantic daily

`lxc launch ubuntu-daily:m`

I've tested a build of the ua-tools in proposed on 22.04's livecd-rootfs, and don't run into the same issue. checking 20.04 as well.

[John Chittum (jchittum)]

I worked up a reproducer of things operating as expected, which makes me believe it's a timing issue around when `livecd-rootfs` and `live-build` do cleanup. on livecd-rootfs:ubuntu/master, i added the following to `live-build/ubuntu-cpc/hooks.d/base/create-root-dir.binary`

env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get --purge remove --assume-yes ubuntu-advantage-tools
env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get autoremove --purge --assume-yes
env DEBIAN_FRONTEND=noninteractive chroot $rootfs_dir apt-get install --assume-yes ubuntu-advantage-tools

I then ran a build of `PROJECT=ubuntu-cpc IMAGE_TARGET=tarball`

afterwards, registered an lxc container locally with the lxd metadata from cloud-images.ubuntu.com (http://cloud-images.ubuntu.com/mantic/20230823/mantic-server-cloudimg-amd64-lxd.tar.xz)

lxc image import mantic-server-cloudimg-amd64-lxd.tar.xz livecd.ubuntu-cpc.rootfs.tar.xz --alias mantic-ua-manual-install

lxc shell immortal-bobcat
root@immortal-bobcat:~# dpkg -l | grep ubuntu-ad
ii ubuntu-advantage-tools 29.3 amd64 management tools for Ubuntu Pro
root@immortal-bobcat:~# ls /etc/apt/preferences.d/
ubuntu-pro-esm-apps ubuntu-pro-esm-infra
root@immortal-bobcat:~# cat /etc/cloud/build.info
build_name: server
serial: 20230901.1405

I'm now going to see if i can work up a reproducer of it _failing_ on Jammy by adding a proposed install during `auto/build` rather than a hook.

[Christian Ehrhardt  (paelzer)]

Since this overall behavior is blocking the scheduled pro release I wanted to make it clear that this is rather important to us. Bumping up priority

[Philip Roche (philroche)]

I have been able to reproduce this issue and create a patch to fix the problem of packaged preferences.d files being removed. I have tested this successfully with mantic base images with ubuntu-advantage-tools 29.3

From changelog history, it has been hard to understand the history of lb_chroot_archives or the intention of the preferences.d files removal. As such, the patch takes the safest approach of not altering the behaviour of removing the packaged preferences.d files, but instead ensures that the packaged preferences.d files are backed up and restored as part of lb-chroot_archives install and remove stages.

[Philip Roche (philroche)]

debdiff attached now too for the mantic release of live-build

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Backup and restore and packaged preferences.d files" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Philip Roche (philroche)]

Apologies. I wasn't very clear. The patches are for the live-build package

[Philip Roche (philroche)]

To aid in testing/reviewing this patch I have performed a qcow2 and a disk-image test build with logs and artifacts @ https://people.canonical.com/~philroche/lp-2033308-live-build/

[Michael Hudson-Doyle (mwhudson)]

So I have a few thoughts/questions here:

1) It would really be better if we could put packaged apt preference files somewhere not in /etc (systemd-style /run-/etc/-/user searching for everything please)

2) Why isn't https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/chroot/100-preserve-apt-prefs.chroot preventing this? That's what it's there for.

3) How many images are out there with this damage? If it's just mantic well whatever but if there are lunar or even older images out there that are missing these files then we'll need to do something in ubuntu-advantage-tools to avoid these prompts on upgrade (I'm not sure what, mind -- we could do hacks to avoid these files being marked as conffiles, which will probably just mean that dpkg stomps on these files on upgrade -- which might be ok? Otherwise it's delicate preinst games I guess).

I think the live-build changes are /probably/ ok. I'll try to meditate on these a little more later. It might be better to use a different backup location than the existing code in livecd-rootfs tries to use? Building a test ubuntu-cpc build is nice but really it's the impact on other project builds that's hard to predict...

[Philip Roche (philroche)]

> 1) It would really be better if we could put packaged apt preference files somewhere not in /etc (systemd-style /run-/etc/-/user searching for everything please)

This would require apt changes though?

> 2) Why isn't https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/chroot/100-preserve-apt-prefs.chroot preventing this? That's what it's there for

This happens too late in the build. See build log https://people.canonical.com/~philroche/lp-2033308-live-build/disk-image-ubuntu-bartender.log . The 010-preserve-apt-prefs.chroot check is at line 34486 of the build log. But debootstrap and lb_chroot_archives removes the files at line 25578 so at the time 010-preserve-apt-prefs.chroot runs there is nothing to backup.

> 3) How many images are out there with this damage? If it's just mantic well whatever but if there are lunar or even older images out there that are missing these files then we'll need to do something in ubuntu-advantage-tools to avoid these prompts on upgrade (I'm not sure what, mind -- we could do hacks to avoid these files being marked as conffiles, which will probably just mean that dpkg stomps on these files on upgrade -- which might be ok? Otherwise it's delicate preinst games I guess).

Mantic only. Only ubuntu-advantage-tools 29 introduced these pinning files as part of addressing https://github.com/canonical/ubuntu-pro-client/issues/2580

[Michael Hudson-Doyle (mwhudson)]

On Mon, 11 Sept 2023 at 23:21, Philip Roche <email address hidden>
wrote:

> > 1) It would really be better if we could put packaged apt preference
> files somewhere not in /etc (systemd-style /run-/etc/-/user searching
> for everything please)
>
> This would require apt changes though?
>

Oh yes. This is just a scream into the void :-)

> > 2) Why isn't https://git.launchpad.net/livecd-rootfs/tree/live-
> build/ubuntu-cpc/hooks.d/chroot/100-preserve-apt-prefs.chroot
> <https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/chroot/100-preserve-apt-prefs.chroot>
> preventing
> this? That's what it's there for
>
> This happens too late in the build. See build log
> https://people.canonical.com/~philroche/lp-2033308-live-build/disk-
> image-ubuntu-bartender.log
> <https://people.canonical.com/~philroche/lp-2033308-live-build/disk-image-ubuntu-bartender.log>
> . The 010-preserve-apt-prefs.chroot check is
> at line 34486 of the build log. But debootstrap and lb_chroot_archives
> removes the files at line 25578 so at the time 010-preserve-apt-
> prefs.chroot runs there is nothing to backup.

Ah OK. Maybe it could be an early chroot hook? Or we just fix it in
live-build as in your patch.

>

> 3) How many images are out there with this damage? If it's just mantic
> well whatever but if there are lunar or even older images out there that
> are missing these files then we'll need to do something in ubuntu-
> advantage-tools to avoid these prompts on upgrade (I'm not sure what,
> mind -- we could do hacks to avoid these files being marked as
> conffiles, which will probably just mean that dpkg stomps on these files
> on upgrade -- which might be ok? Otherwise it's delicate preinst games I
> guess).
>
> Mantic only. Only ubuntu-advantage-tools 29 introduced these pinning
> files as part of addressing https://github.com/canonical/ubuntu-pro-
> client/issues/2580
> <https://github.com/canonical/ubuntu-pro-client/issues/2580>

OK. That's a relief :-)

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu49

---------------
live-build (3.0~a57-1ubuntu49) mantic; urgency=medium

  * To avoid removing packaged preferences.d files, in lb_chroot_archives install stage,
    only remove apt/preferences.d/* files after we backup any existing preferences.d/ files.
    Then during remove stage or lb_chroot_archives restore the backed up packaged
    preferences.d files. LP: #2033308

 -- Phil Roche <email address hidden> Tue, 12 Sep 2023 22:36:21 +1200

[Philip Roche (philroche)]

This fix will need to be backported all the way back to xenial's version of live-build. Can you update the bug metadata/Affects to reflect this?

[Philip Roche (philroche)]

Lunar backport debdiff

[Philip Roche (philroche)]

debdiff for lunar backport attached

[Catherine Redfield (catred)]

For testing that the patch works as expected, we applied a testing patch to live-build which adds a dummy preferences.d conf file. We then created an image using the test-patch modified live-build and verified that the dummy preferences file existed in the resulting image.

[Catherine Redfield (catred)]

Testing patch for jammy live-build attached.

[Catherine Redfield (catred)]

Attached jammy debdiff and testing info

[Catherine Redfield (catred)]

Focal backport debdiff

[Catherine Redfield (catred)]

Testing patch for focal

[Catherine Redfield (catred)]

debdiff and testing patch attached

[Catherine Redfield (catred)]

debdiff for bionic

[Catherine Redfield (catred)]

Testing patch for bionic

[Catherine Redfield (catred)]

debdiff and testing patch attached

[Catherine Redfield (catred)]

debdiff for xenial

[Catherine Redfield (catred)]

testing patch for xenial (all the testing patches are the same but including this final one for completeness)

[Catherine Redfield (catred)]

debdiff and testing patch applied

[Andreas Hasenack (ahasenack)]

Hi, I have some comments/questions.

a) In the test plan, this command doesn't work:

  # Create an image using live-build
  $ make disk-image-minimized

Does this need some special configuration, or branch? I presumed this was to be run in the root of the source package, after applying all patches (including the test patch).

b) given the uncertainty around the motivation for the original removal of all preferences files during the build, have you tested building an image with and without the fix, and compared them? Could this be added to the test plan, to make sure the fix is not inadvertently introducing other changes?

c) I'm not super familiar with live-build. I don't know if you can "cross-build" an image, i.e., host is jammy, and you build an image for focal. In other words, the test plan needs to be executed on each ubuntu release, and the image target needs to be that same OS release, is that correct?

[Andreas Hasenack (ahasenack)]

d) regarding saving the preferences backup in etc/apt/preferences.d.save, and the "risk" of not cleaning up and leaving this directory there, why not use a temporary directory, or even preferences.d.save in /tmp? Not a blocker, but I was reading your regression potential analysis, and wondered. Maybe it doesn't fit well with how all these scripts work together?

[Andreas Hasenack (ahasenack)]

e) Looking at the code, and you highlighted this in the regression section, we see:
        # Check local pinning preferences
        if Find_files config/archives/*.pref || \
           Find_files config/archives/*.pref.chroot
        then
            for FILE in config/archives/*.pref \
                    config/archives/*.pref.chroot
            do
                if [ -e "${FILE}" ]
                then
                    cp ${FILE} chroot/etc/apt/preferences.d/$(basename ${FILE} .chroot)
                fi
            done
        fi

So there is some sort of preferences.d file handling in place already, for customizations during build (and this is what we could overwrite if the backed-up preferences file have the same name). Are we making use of this in any place in our regular builds?

[Philip Roche (philroche)]

> e) Looking at the code, and you highlighted this in the regression section

This is only applicable for preferences configured in the `config/archives/` directory prior to image build. This is not a code path for packages apt preferences.

We are not using `config/archives/*.pref` or `config/archives/*.pref.chroo` in any of our cloud image builds.

[Philip Roche (philroche)]

> d) regarding saving the preferences backup in etc/apt/preferences.d.save, and the "risk" of not cleaning up and leaving this directory there

A good point. This was following the same pattern as the other `/etc/apt/preferences.d` backup and restore login.

There are multiple places in live-build which clean up any `/etc/apt/preferences.d.save` directory before build is complete.

[Philip Roche (philroche)]

> c) I'm not super familiar with live-build. I don't know if you can "cross-build" an image, i.e., host is jammy, and you build an image for focal. In other words, the test plan needs to be executed on each ubuntu release, and the image target needs to be that same OS release, is that correct?

Correct. They need to be the same. I am not aware of any cases of cross building.

[Philip Roche (philroche)]

> b) given the uncertainty around the motivation for the original removal of all preferences files during the build, have you tested building an image with and without the fix, and compared them? Could this be added to the test plan, to make sure the fix is not inadvertently introducing other changes?

We have tested with before and after fix yes and compared the filelists.

@catred is working to add this to the test plan

[Catherine Redfield (catred)]

> a) Does [the make command] need some special configuration, or branch? I presumed this was to be run in the root of the source package, after applying all patches (including the test patch).

My apologies, the make command was run in the cloud image build repo but it is a pass-through from livecd-rootfs and does not require a specific directory -- the command it runs is ubuntu-bartender. I have updated the Test Plan to reflect this more universally accessible command.

[Catherine Redfield (catred)]

> d) regarding saving the preferences backup in etc/apt/preferences.d.save, and the "risk" of not cleaning up and leaving this directory there

Since /etc/apt/preferences.d.save is what is used in the mantic live-build, I feel like it makes most sense to keep the same temporary folder on backport to the older releases. I also experimented this morning with using /tmp instead of /etc/apt and the restore does not work; it seems that something between the backup and restore clears out /tmp.

> b) given the uncertainty around the motivation for the original removal of all preferences files during the build, have you tested building an image with and without the fix, and compared them? Could this be added to the test plan, to make sure the fix is not inadvertently introducing other changes?

I added notes for building patched and unpatched images to the Test Plan and comparing filelists and manifests.

[Andreas Hasenack (ahasenack)]

Thank you all for the replies, and test case updates.

[Andreas Hasenack (ahasenack)]

Hello Dan, or anyone else affected,

Accepted live-build into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu44.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Dan, or anyone else affected,

Accepted live-build into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu41.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Dan, or anyone else affected,

Accepted live-build into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu38.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

What's the plan for xenial and bionic? SRU, ou ESM?

[Catherine Redfield (catred)]

> What's the plan for xenial and bionic? SRU, ou ESM?

ESM. We are waiting to hear back from the security team to upload it to their queue (hopefully soon).

[Philip Roche (philroche)]

> ESM. We are waiting to hear back from the security team to upload it to their queue (hopefully soon).

This is assuming we can consume live-build from ESM during builds. Have we confirmed this assumption?

Based on a previous GCE bionic Pro image build @ https://launchpad.net/~cloudware/+livefs/ubuntu/bionic/cpc/+build/518339 (only visible to ~cloudware members) live-build is from the regular archive and not ESM. I have not looked in to if it is possible to consume from ESM at this stage in a build.

[Catherine Redfield (catred)]

LUNAR

# Pull unpatched live-build
catred@canonical:~/code/live-build$ git status
On branch applied/ubuntu/lunar
Your branch is up to date with 'origin/applied/ubuntu/lunar'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
 lb_chroot_archives-preferences_test.patch

nothing added to commit but untracked files present (use "git add" to track)
catred@canonical:~/code/live-build$ quilt import lb_chroot_archives-preferences_test.patch
Importing patch lb_chroot_archives-preferences_test.patch (stored as patches/lb_chroot_archives-preferences_test.patch)
catred@canonical:~/code/live-build$ quilt push
Applying patch patches/lb_chroot_archives-preferences_test.patch
patching file scripts/build/lb_chroot
patching file scripts/build/lb_chroot_create_preferences

Now at patch patches/lb_chroot_archives-preferences_test.patch

# Create an image using unpatched live-build
catred@canonical:~/builds/live-build-3057/patch_validation/lunar$ export LIVECD_ROOTFS_DIR="/home/catred/code/livecd-rootfs"
catred@canonical:~/builds/live-build-3057/patch_validation/lunar$ export LIVECD_ROOTFS_LOCAL_DIR="/home/catred/code/livecd-rootfs"
catred@canonical:~/builds/live-build-3057/patch_validation/lunar$ export LIVE_BUILD_DIR="/home/catred/code/live-build"
catred@canonical:~/builds/live-build-3057/patch_validation/lunar$ ubuntu-bartender --build-provider aws --livecd-rootfs-branch ubuntu/lunarmaster -- --series lunar --project ubuntu-cpc --image-target disk-image --subproject minimized
Sourcing configuration file: /home/catred/.ubuntu-bartender.rc
Creating fair-shiner-ubuntu-bartender on AWS... with profile default in region us-east-1 using catred-cpc-jenkins keypair
Preparing ingredients... in /home/catred/ubuntu-bartender-multipass.ph1jlklwyQ
LIVECD_ROOTFS_DIR is set, copying /home/catred/code/livecd-rootfs to /home/catred/ubuntu-bartender-multipass.ph1jlklwyQ/livecd-rootfs ...
LIVE_BUILD_DIR is set, copying /home/catred/code/live-build to /home/catred/ubuntu-bartender-multipass.ph1jlklwyQ/live-build ...
UBUNTU_OLD_FASHIONED_DIR is set, copying /home/catred/code/ubuntu-old-fashioned to /home/catred/ubuntu-bartender-multipass.ph1jlklwyQ/ubuntu-old-fashioned ...
uploading ingredients.tar.gz to fair-shiner-ubuntu-bartender ...
extracting ingredients.tar.gz in fair-shiner-ubuntu-bartender ...
remove ingredients.tar.gz in fair-shiner-ubuntu-bartender as it is no longer required ...
removed 'ingredients.tar.gz'
Mixing drink...
(See progress in fair-shiner-ubuntu-bartender.log)
Pouring fair-shiner-ubuntu-on-the-rocks.tar.gz...
Cleaning up...
Destroying fair-shiner-ubuntu-bartender on AWS...
catred@canonical:~/builds/live-build-3057/patch_validation/lunar$ grep 'Creating dummy preferences' fair-shiner-ubuntu-bartender.log
+ Echo_message Creating dummy preferences.d files...
+ STRING=Creating dummy preferences.d files...
P:+ printf Creating dummy preferences.d files...\n
 Creating dummy preferences.d files...

# Download and extract the patched live-build
# Add testing patch to live-build so behaviour is apparent
catred@canonical:~/packaging/cpc3057backport/lunar/live-build$ quilt import ~/code/live-build/lb_chroot_archives-preferences_te...

[Catherine Redfield (catred)]

JAMMY

# Pull unpatched live-build
catred@canonical:~/code/live-build$ git status
On branch applied/ubuntu/jammy
Your branch is up to date with 'origin/applied/ubuntu/jammy'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
 lb_chroot_archives-preferences_test.patch

nothing added to commit but untracked files present (use "git add" to track)
catred@canonical:~/code/live-build$ quilt import lb_chroot_archives-preferences_test.patch
Importing patch lb_chroot_archives-preferences_test.patch (stored as patches/lb_chroot_archives-preferences_test.patch)
catred@canonical:~/code/live-build$ quilt push
Applying patch patches/lb_chroot_archives-preferences_test.patch
patching file scripts/build/lb_chroot
patching file scripts/build/lb_chroot_create_preferences

Now at patch patches/lb_chroot_archives-preferences_test.patch

# Create an image using unpatched live-build
catred@canonical:~/builds/live-build-3057/patch_validation/jammy$ export LIVECD_ROOTFS_DIR="/home/catred/code/livecd-rootfs"
catred@canonical:~/builds/live-build-3057/patch_validation/jammy$ export LIVECD_ROOTFS_LOCAL_DIR="/home/catred/code/livecd-rootfs"
catred@canonical:~/builds/live-build-3057/patch_validation/jammy$ export LIVE_BUILD_DIR="/home/catred/code/live-build"
catred@canonical:~/builds/live-build-3057/patch_validation/jammy$ ubuntu-bartender --build-provider aws --livecd-rootfs-branch ubuntu/jammymaster -- --series jammy --project ubuntu-cpc --image-target disk-image --subproject minimized
Sourcing configuration file: /home/catred/.ubuntu-bartender.rc
Creating noted-bream-ubuntu-bartender on AWS... with profile default in region us-east-1 using catred-cpc-jenkins keypair
Preparing ingredients... in /home/catred/ubuntu-bartender-multipass.YZoZUnxwKb
LIVECD_ROOTFS_DIR is set, copying /home/catred/code/livecd-rootfs to /home/catred/ubuntu-bartender-multipass.YZoZUnxwKb/livecd-rootfs ...
LIVE_BUILD_DIR is set, copying /home/catred/code/live-build to /home/catred/ubuntu-bartender-multipass.YZoZUnxwKb/live-build ...
UBUNTU_OLD_FASHIONED_DIR is set, copying /home/catred/code/ubuntu-old-fashioned to /home/catred/ubuntu-bartender-multipass.YZoZUnxwKb/ubuntu-old-fashioned ...
uploading ingredients.tar.gz to noted-bream-ubuntu-bartender ...
extracting ingredients.tar.gz in noted-bream-ubuntu-bartender ...
remove ingredients.tar.gz in noted-bream-ubuntu-bartender as it is no longer required ...
removed 'ingredients.tar.gz'
Mixing drink...
(See progress in noted-bream-ubuntu-bartender.log)
Pouring noted-bream-ubuntu-on-the-rocks.tar.gz...
Cleaning up...
Destroying noted-bream-ubuntu-bartender on AWS...
catred@canonical:~/builds/live-build-3057/patch_validation/jammy$ grep 'Creating dummy preferences' noted-bream-ubuntu-bartender.log
+ Echo_message Creating dummy preferences.d files...
+ STRING=Creating dummy preferences.d files...
P:+ printf Creating dummy preferences.d files...\n
 Creating dummy preferences.d files...

# Download and extract the patched live-build
# Add testing patch to live-build so behaviour is apparent
catred@canonical:~/packaging/cpc3057backport/jammy/live-build$ quilt import ~/code/live-build/lb_chroot_archives-preferences_test.pat...

[Catherine Redfield (catred)]

FOCAL

# Pull unpatched live-build
catred@canonical:~/code/live-build$ git status
On branch applied/ubuntu/focal
Your branch is up to date with 'origin/applied/ubuntu/focal'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
 lb_chroot_archives-preferences_test.patch

nothing added to commit but untracked files present (use "git add" to track)
catred@canonical:~/code/live-build$ quilt import lb_chroot_archives-preferences_test.patch
Importing patch lb_chroot_archives-preferences_test.patch (stored as patches/lb_chroot_archives-preferences_test.patch)
catred@canonical:~/code/live-build$ quilt push
Applying patch patches/lb_chroot_archives-preferences_test.patch
patching file scripts/build/lb_chroot
patching file scripts/build/lb_chroot_create_preferences

Now at patch patches/lb_chroot_archives-preferences_test.patch

# Create an image using unpatched live-build
catred@canonical:~/builds/live-build-3057/patch_validation/focal$ export LIVECD_ROOTFS_DIR="/home/catred/code/livecd-rootfs"
catred@canonical:~/builds/live-build-3057/patch_validation/focal$ export LIVECD_ROOTFS_LOCAL_DIR="/home/catred/code/livecd-rootfs"
catred@canonical:~/builds/live-build-3057/patch_validation/focal$ export LIVE_BUILD_DIR="/home/catred/code/live-build"
catred@canonical:~/builds/live-build-3057/patch_validation/focal$ ubuntu-bartender --build-provider aws --livecd-rootfs-branch ubuntu/focalmaster -- --series focal --project ubuntu-cpc --image-target disk-image --subproject minimized
Sourcing configuration file: /home/catred/.ubuntu-bartender.rc
Creating ruling-imp-ubuntu-bartender on AWS... with profile default in region us-east-1 using catred-cpc-jenkins keypair
Preparing ingredients... in /home/catred/ubuntu-bartender-multipass.ZG9BxVoBmL
LIVECD_ROOTFS_DIR is set, copying /home/catred/code/livecd-rootfs to /home/catred/ubuntu-bartender-multipass.ZG9BxVoBmL/livecd-rootfs ...
LIVE_BUILD_DIR is set, copying /home/catred/code/live-build to /home/catred/ubuntu-bartender-multipass.ZG9BxVoBmL/live-build ...
UBUNTU_OLD_FASHIONED_DIR is set, copying /home/catred/code/ubuntu-old-fashioned to /home/catred/ubuntu-bartender-multipass.ZG9BxVoBmL/ubuntu-old-fashioned ...
uploading ingredients.tar.gz to ruling-imp-ubuntu-bartender ...
extracting ingredients.tar.gz in ruling-imp-ubuntu-bartender ...
remove ingredients.tar.gz in ruling-imp-ubuntu-bartender as it is no longer required ...
removed 'ingredients.tar.gz'
Mixing drink...
(See progress in ruling-imp-ubuntu-bartender.log)
Pouring ruling-imp-ubuntu-on-the-rocks.tar.gz...
Cleaning up...
Destroying ruling-imp-ubuntu-bartender on AWS...
catred@canonical:~/builds/live-build-3057/patch_validation/focal$ grep 'Creating dummy preferences' ruling-imp-ubuntu-bartender.log
+ Echo_message Creating dummy preferences.d files...
+ STRING=Creating dummy preferences.d files...
P:+ printf Creating dummy preferences.d files...\n
 Creating dummy preferences.d files...

# Download and extract the patched live-build
# Add testing patch to live-build so behaviour is apparent
catred@canonical:~/packaging/cpc3057backport/focal/live-build$ quilt import ~/code/live-build/lb_chroot_archives-preferences_test.patch
Imp...

[Catherine Redfield (catred)]

Since xenial and bionic are EOL, those patches will not included in the archive or ESM. We will be uploading to a private PPA for image builds.

[Catherine Redfield (catred)]

Zipped archives of the test images (patched and unpatched) can be found here: https://people.canonical.com/~catred/lp-2033308-validation/

[Catherine Redfield (catred)]

Verification tags updated

[Andreas Hasenack (ahasenack)]

I sat down with @catred to discuss how the verification was performed, because there was no indication of which live-build package she used for the before and after tests, and where they came from. She still had the test environment untouched, so we were able to verify that the correct versions were used.

For future SRUs of live-build, should they happen, we agreed on a less ambiguous verification, which would make this information clear.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu44.1

---------------
live-build (3.0~a57-1ubuntu44.1) lunar; urgency=medium

  * To avoid removing packaged preferences.d files, in lb_chroot_archives
    install stage, only remove apt/preferences.d/* files after we backup
    any existing preferences.d/ files. Then during remove stage or
    lb_chroot_archives restore the backed up packaged preferences.d files.
    (LP: #2033308)

 -- Philip Roche <email address hidden> Tue, 10 Oct 2023 17:29:22 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for live-build has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu41.22.04.1

---------------
live-build (3.0~a57-1ubuntu41.22.04.1) jammy; urgency=medium

  * To avoid removing packaged preferences.d files, in lb_chroot_archives
    install stage, only remove apt/preferences.d/* files after we backup
    any existing preferences.d/ files. Then during remove stage or
    lb_chroot_archives restore the backed up packaged preferences.d files.
    (LP: #2033308)

 -- Catherine Redfield <email address hidden> Wed, 18 Oct 2023 10:58:23 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu38.20.04.3

---------------
live-build (3.0~a57-1ubuntu38.20.04.3) focal; urgency=medium

  * To avoid removing packaged preferences.d files, in lb_chroot_archives
    install stage, only remove apt/preferences.d/* files after we backup
    any existing preferences.d/ files. Then during remove stage or
    lb_chroot_archives restore the backed up packaged preferences.d files.
    (LP: #2033308)

 -- Catherine Redfield <email address hidden> Wed, 18 Oct 2023 12:16:49 -0400

[Catherine Redfield (catred)]

Since xenial and bionic are not of general interest (no one is really building these images any more), the patches to those branches will not be pushed to ESM updates. If anyone is interested in using a patched version of live-build, the patched code can be found here: https://launchpad.net/~cloud-images/cloud-images/+git/live-build