Any packages installed in binary hooks are marked as manually installed - even dependencies

[Philip Roche]

After much debugging trying to understand why `apt-get autoremove` was not working in a binary hook I realised that all packages installed in a binary hook are marked as manually installed - even the dependencies of packages. This is a bug.

Another side affect of this is that any dependency of a package in the base chroot that is removed in a binary hook is not autoremovable in the binary hook.

It is autoremovable once the image is created and launched though.

This is due to how the `setup_mountpoint` function works (see https://git.launchpad.net/livecd-rootfs/tree/live-build/functions#n123) specifically

```
mount -t tmpfs none "$mountpoint/var/lib/apt"
```

Which means that inside the chroot that the binary hook works on, the `/var/lib/apt` directory is empty... including the `/var/lib/apt/extended_states` file which is used to store whether a package was auto installed or not.

The same goes the other way - any changes to the `/var/lib/apt/extended_states` file inside chroot that the binary hook is changing are not reflected in the final image due to the `teardown_mountpoint` (see https://git.launchpad.net/livecd-rootfs/tree/live-build/functions#n152) not persisting any of those changes.

The end result being.

* If a package is removed in the binary hook then the resulting image will likely have autoremovable packages at first launch
* Any package and their dependencies installed in a binary hook will all be marked as manually installed.

SRU
====

[ Impact ]

 * Packages installed in a binary hook are marked as manually installed - even the dependencies of packages

[ Test Plan ]

 * Build test images and remove a package with dependencies.
   * Confirm that the dependencies are marked as autoremovable.
   * Confirm there are packages marked as auto installed

[ Where problems could occur ]

 * Is any dependent packages are not marked as autremovable.
 * If no packages are marked as auto installed

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 23.10.39

---------------
livecd-rootfs (23.10.39) mantic; urgency=medium

  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    LP: #2036195.

 -- Steve Langasek <email address hidden> Mon, 18 Sep 2023 15:06:23 -0700

[Philip Roche (philroche)]

This fix will need to be backported all the way back to xenial's version of livecd-rootfs. Can you update the bug metadata/Affects to reflect this?

[Steve Langasek (vorlon)]

Done, but note that the bug state does not actually have to be adjusted prior to SRU upload.

[Philip Roche (philroche)]

Lunar backport @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/454058

[Philip Roche (philroche)]

Jammy backport @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/454065

[Philip Roche (philroche)]

Focal backport @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/454066

[Philip Roche (philroche)]

Bionic backport https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/454067

[Philip Roche (philroche)]

Xenial backport @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/454068

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to jammy-proposed has been rejected from the upload queue for the following reason: "cruft in upload (.idea*/)".

[Philip Roche (philroche)]

Apologies RE jammy upload. I have now re-uploaded Jammy livecd-rootfs 2.765.30 again without any '.idea' directory cruft.

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to lunar-proposed has been rejected from the upload queue for the following reason: ".idea cruft, needs reuploaded".

[Philip Roche (philroche)]

As there will be no further uploads accepted for xenial and bionic the proposed changes @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+ref/bugfix/ubuntu-cpc-LP-2036195-auto-installed-packages-xenial for xenial and https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+ref/bugfix/ubuntu-cpc-LP-2036195-auto-installed-packages-bionic for bionic have been merged to https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+ref/ubuntu/xenial and https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+ref/ubuntu/bionic respectively, but there will be no corresponding package upload.

[Philip Roche (philroche)]

Apologies RE lunar upload. I have now re-uploaded Lunar livecd-rootfs 2.828.2 again without any '.idea' directory cruft - see unapproved queue @ https://launchpad.net/ubuntu/lunar/+queue?queue_state=1&queue_text=livecd-rootfs

[Philip Roche (philroche)]

I have re-uploaded focal livecd-rootfs too 2.664.51 without any '.idea' directory cruft. See unapproved queue @ https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=livecd-rootfs

[Andreas Hasenack (ahasenack)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.828.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.51 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted livecd-rootfs (2.664.51) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.664.51 (arm64)
ubuntu-image/1.11+20.04ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Philip Roche (philroche)]

I have created test images for focal, jammy and lunar using this proposed package.

This is easiest to test using a GCE image build where image build removes fwupd, fwupd-signed, bolt, libarchive13 and irqbalance which then leaves a long list of package autoremovable.

```
Remv jq [1.6-2.1ubuntu3]
Remv libatasmart4 [0.19-5build2]
Remv libblockdev-fs3 [3.0.2-3]
Remv libblockdev-loop3 [3.0.2-3]
Remv libblockdev-nvme3 [3.0.2-3]
Remv libblockdev-part3 [3.0.2-3]
Remv libblockdev-swap3 [3.0.2-3]
Remv libblockdev3 [3.0.2-3]
Remv libblockdev-utils3 [3.0.2-3]
Remv libflashrom1 [1.3.0-2.1ubuntu1]
Remv libftdi1-2 [1.5-6build2]
Remv libjq1 [1.6-2.1ubuntu3]
Remv libqmi-proxy [1.32.4-2ubuntu1]
Remv libqmi-glib5 [1.32.4-2ubuntu1]
Remv libmbim-utils [1.28.4-2]
Remv libmbim-glib4 [1.28.4-2] [libmbim-proxy:amd64 ]
Remv libmbim-proxy [1.28.4-2]
Remv libmm-glib0 [1.20.6-1ubuntu1]
Remv libnss3 [2:3.92-1]
Remv libnspr4 [2:4.35-1.1]
Remv libnuma1 [2.0.16-1]
Remv libonig5 [6.9.8-2]
Remv libprotobuf-c1 [1.4.1-1ubuntu2]
Remv usb-modeswitch [2.6.1-3ubuntu2]
Remv tcl [8.6.13]
Remv tcl8.6 [8.6.13+dfsg-2]
Remv libtcl8.6 [8.6.13+dfsg-2]
Remv libudisks2-0 [2.10.0-3ubuntu2]
Remv usb-modeswitch-data [20191128-5]
```

Using the proposed packages I can confirm that after calling `apt-get -y autoremove --purge` that these packages are no longer present in the image or the package manifest for the image.

Marking verification-done

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.828.3

---------------
livecd-rootfs (2.828.3) lunar; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

livecd-rootfs (2.828.2) lunar; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.828.1) lunar; urgency=medium

  * Fix unminimize to correctly list packages. (LP: #1996489)
  * Install LXD snap from stable/ubuntu-<version> channel. (LP: #2036725)

 -- Steve Langasek <email address hidden> Thu, 07 Dec 2023 08:51:51 -0800

[Łukasz Zemczak (sil2100)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Philip Roche (philroche)]

As per https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2036195/comments/19 this change has been successfully verified for Jammy

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.34

---------------
livecd-rootfs (2.765.34) jammy; urgency=medium

  * Remove comments in the debian/control dependencies as it was causing the
    package build to fail.

livecd-rootfs (2.765.33) jammy; urgency=medium

  [ Steve Langasek ]
  * Use losetup instead of kpartx to resolve race conditions in riscv64
    image builds. LP: #2045797.

  [ Utkarsh Gupta ]
  * unminimize: Use lxd-installer to install LXD itself (LP: #2036725)

livecd-rootfs (2.765.32) jammy; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

livecd-rootfs (2.765.31) jammy; urgency=medium

  [ John Chittum ]
  * fix: add 6.5 kernel apparmor features to livecd-rootfs based on
    features of 6.5 in ubuntu/mantic. This will roll as HWE. (LP: #2037567)

livecd-rootfs (2.765.30) jammy; urgency=medium

  * Enable snap preseeding with ppc64el images where /boot/vmlinux is used
    instead of /boot/vmlinuz. (LP: #2038957)

livecd-rootfs (2.765.29) jammy; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.765.28) jammy; urgency=medium

  * Fix unminimize to correctly list packages. (LP: #1996489)
  * Install LXD snap from stable/ubuntu-<version> channel. (LP: #2036725)

livecd-rootfs (2.765.27) jammy; urgency=medium

  [ Thomas Bechtold ]
  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds (LP: #1968873)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 15 Jan 2024 16:41:18 +0100

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.52 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Philip Roche (philroche)]

I have verified this change once more with test build using livecd-rootfs 2.664.52 in focal-proposed.

Any dependent packages on packages removed during build were successfully removed as they were successfully marked as autoremovable. Also confirmed that any dependent packages also marked auto auto installed.

Marking as `verification-done-focal`

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.664.52

---------------
livecd-rootfs (2.664.52) focal; urgency=medium

  * fix: use correct sshd_config.d/ ordering. (LP: #2049860)

livecd-rootfs (2.664.51) focal; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.664.50) focal; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds. (LP: #1968873)

 -- Thomas Bechtold <email address hidden> Mon, 22 Jan 2024 17:08:05 +0530