Only keep 2 kernels

[Julian Andres Klode]

[Impact]
APT currently keeps 3 kernels or even 4 in some releases. Our boot partition is sized for a steady state of 2 kernels + 1 new one being unpacked, hence users run out of space and new kernels fail to install, upgrade runs might abort in the middle. It's not nice.

[Test plan]
1. Have two kernels installed (let's call them version 3, 2)
2. Check that both kernels are not autoremovable
3. Install an old kernel (let's call it 1), and mark it automatic
4. Check that 1 will be autoremovable (apt autoremove -s)
5. Reboot into 1, check that 2 is autoremovable (apt autoremove -s)
6. Actually remove 2
7. Reboot into 3 and check that both 1 and 3 are now not autoremovable

[Where problems could occur]
We could keep the wrong kernels installed that the user did not expect.

We remove the requirement to keep the most recently installed version, previously recorded in APT::LastInstalledKernel, to achieve this, as we had 3 hard requirements so far:

1. keep booted kernel
2. keep highest version
3. keep most recently installed

1 can't be removed as it would break running systems, 2 is what you definitely want to keep.

During normal system lifetime, the most recently installed kernel is the same as the highest version, so 2==3, and there are no changes to behavior.

Likewise, if you most recently installed an older kernel manually for debugging, it would be manually installed and not subject to removal, even if the rule is dropped.

The behavior really only changes if you install an older kernel, and then mark it auto - that older kernel becomes automatically removable immediately after it is marked as auto.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.4.5

---------------
apt (2.4.5) unstable; urgency=medium

  * Only protect two kernels, not last installed one (LP: #1968154)
  * Fix segfault in CacheSetHelperAPTGet::tryVirtualPackage()

 -- Julian Andres Klode <email address hidden> Fri, 08 Apr 2022 12:22:23 +0200

[Julian Andres Klode (juliank)]

unattended-upgrades in jammy did not need a change, we'll see if SRUs will need changes, hence keeping the task so we can add them later on without having to reset to launchpad API shenanigans.

[Łukasz Zemczak (sil2100)]

Can you mention a bit more about the APT::LastInstalledKernel logic usage removal in this SRU? What was it used before? Since right now this SRU seems to be dropping its use + only keeping track of 2 kernels, can you help me understand the implications of that (as I didn't know about this logic before)?

[Julian Andres Klode (juliank)]

@sil2100 Added that to regression potential. It was used to keep the third kernel. There was some misunderstanding how that affects the set calculated.

I noticed that 2.0.7 and 2.3.9ubuntu0.1 had the wrong version in maintainer scripts, so I will have to upload a fixed 2.0.8 for the former (it has proper upstream release, and possibly other downstreams), and replace 2.3.9ubuntu0.1 with a newer one.

[Timo Aaltonen (tjaalton)]

Hello Julian, or anyone else affected,

Accepted apt into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.3.9ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Julian, or anyone else affected,

Accepted apt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.0.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Julian, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.3.9ubuntu0.1) for impish have finished running.
The following regressions have been reported in tests triggered by the package:

reprotest/0.7.16 (s390x, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.0.8) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

reprotest/0.7.14 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.6.15) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

autopkgtest/5.3.1ubuntu1.1 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

Verified impish 2.3.9ubuntu0.1

Log:
1. (in 1016) started with 5.13.0-1016-kvm; installed 5.13.0-1022-kvm
2. (in 1016) no kernels to autoremove
3. (in 1016) installed linux-image-5.13.0-1011-kvm linux-modules-5.13.0-1011-kvm, and marked auto
4. (in 1016) SUCCESS: autoremove would remove 1011
5. (in 1011) SUCCESS: After reboot into 1011, 1016 is autoremovable (*1)
7. (in 1022) SUCCESS: After reboot into 1022, both 1011 and 1022 are kept

(*1) There is a bug in the image used by lxd: The 1016 kernel that was preinstalled on the image was marked as manually installed, this was corrected with apt-mark auto linux-.*-5.13.0-1016-kvm

[Jarno Suni (jarnos)]

Test passed for apt 2.0.8

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.3.9ubuntu0.1

---------------
apt (2.3.9ubuntu0.1) impish; urgency=medium

  * Only protect two kernels, not last installed one (LP: #1968154)
  * Point to impish in gitlab-ci and gbp.conf

 -- Julian Andres Klode <email address hidden> Mon, 25 Apr 2022 16:14:41 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.0.8

---------------
apt (2.0.8) focal; urgency=medium

  * Adjust conffile removal version in postinst and maintscript; followup
    fix for (LP: #1968154)

apt (2.0.7) focal; urgency=medium

  * Revert "Protect currently running kernel at run-time"
  * Backport Determine autoremovable kernels at run-time (LP: #1615381) as of
    2.4.5; including the change to only protect two kernels, not last installed
    one (LP: #1968154)

 -- Julian Andres Klode <email address hidden> Mon, 25 Apr 2022 15:58:46 +0200

[David Röthlisberger (drothlis)]

This is causing me trouble building a focal (20.04) userspace from scratch:

    $ dpkg --configure -a
    [...]
    Setting up apt (2.0.8) ...
    /var/lib/dpkg/info/apt.postinst: 65: /etc/kernel/postinst.d/apt-auto-removal: not found
    dpkg: error processing package apt (--configure):
     installed apt package post-installation script subprocess returned error exit status 127

It works with apt 2.0.6.

I think it's because the patch[1] removes `/etc/kernel/postinst.d/apt-auto-removal`:

    diff --git a/debian/rules b/debian/rules
    index 7997739..8a110f7 100755
    --- a/debian/rules
    +++ b/debian/rules
    @@ -29,7 +29,6 @@ override_dh_install-arch:
      dh_install -papt -Xmethods/curl -Xmethods/curl+https -Xmethods/curl+http
      dh_install --remaining
      install -m 644 debian/apt.conf.autoremove debian/apt/etc/apt/apt.conf.d/01autoremove
    - install -m 755 debian/apt.auto-removal.sh debian/apt/etc/kernel/postinst.d/apt-auto-removal

     override_dh_gencontrol:
      dh_gencontrol -- -Vapt:keyring="$(shell ./vendor/getinfo keyring-package)"

...but `apt.postinst` still contains this:

    # create kernel autoremoval blacklist on update
    if dpkg --compare-versions "$2" lt 0.9.9.3; then
        /etc/kernel/postinst.d/apt-auto-removal
    fi

Since I'm building this from scratch, "$2" (the most-recently-configured-version) is empty, and `dpkg --compare-versions lt` returns true.

[1]: https://git.launchpad.net/ubuntu/+source/apt/commit/?id=f9d2d993687c0d5223c241956ef6a0aabcf15bf0

[Julian Andres Klode (juliank)]

Please create a new bug for that regression, thanks for noticing!

[David Röthlisberger (drothlis)]

Raised #1974456.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Julian, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (1.6.17) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

unattended-upgrades/1.1ubuntu1.18.04.14 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

bionic: apt 1.6.17, Started with linux-image-4.15.0-208-generic

Log:
1. (in 208) installed 206
2. (in 208) no kernels to autoremove
3. (in 208) installed 204
4. (in 208) SUCCESS: autoremove would remove 204
5. (in 204) SUCCESS: After reboot into 204, 206 is autoremovable and 204, 208 are kept
6. (in 208 again) 204 is offered for removal, but I remove 206 per instructions
7. (in 208) SUCCESS: Both the remaining 204 and the 208 are kept.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.6.17

---------------
apt (1.6.17) bionic; urgency=medium

  * Bump cache minor version for kernel collection.
    The previous update introduced the new kernel autoremoval code which
    collects kernels in the cache as providers of a $kernel package.
    We need to bump the cache minor version for this as otherwise the
    $kernel package might not exist and all kernels end up autoremovable
    until a package got installed or sources updated.

apt (1.6.16) bionic; urgency=medium

  * postinst: Do not run removed kernel postinst.d script (LP: #1974456)

apt (1.6.15) bionic; urgency=medium

  * Revert "Protect currently running kernel at run-time"
  * Backport Determine autoremovable kernels at run-time (LP: #1615381) as of
    2.4.5; including the change to only protect two kernels, not last installed
    one (LP: #1968154)

 -- Julian Andres Klode <email address hidden> Fri, 12 Aug 2022 12:38:33 +0200

[Alex Kompel (velocloud)]

There is a regression w.r.t. unsigned kernel packages:

- install linux-image-unsigned-5.4.0-1099-aws, reboot
- install linux-image-unsigned-5.4.0-1100-aws
- mark both as auto
- autoremove will attempt to remove 1100

apt 1.6.14 would keep both.

[Julian Andres Klode (juliank)]

If those are your only two kernels both should be kept. If you have a higher versioned kernel, it's correct that 1100 is removed. That's a feature, not a bug.

Also don't do that.

[Alex Kompel (velocloud)]

Don't to what? Install unsigned kernel packages? My point is that there is a regression. linux-image-unsigned-* are longer considered kernel packages with the new patch.

If linux-image-unsigned-5.4.0-1099-aws and linux-image-unsigned-5.4.0-1100-aws were the only kernel packages on the system, they both would be removed with 1.16.17 including the active kernel. That was not the case in 1.16.14.

# apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  linux-image-unsigned-5.4.0-1099-aws linux-image-unsigned-5.4.0-1100-aws linux-modules-5.4.0-1099-aws linux-modules-5.4.0-1100-aws
0 upgraded, 0 newly installed, 4 to remove and 66 not upgraded.
After this operation, 165 MB disk space will be freed.
Do you want to continue? [Y/n] ^C

# uname -a
Linux ip-xxx 5.4.0-1100-aws #108~18.04.1-Ubuntu SMP Thu Mar 30 02:15:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[bmaupin (bmaupin)]

Am I running into this bug or a different one?

apt dist-upgrade started failing for me today. In particular, linux-firmware is failing to install because /boot is full.

But it looks like I have 4 kernels installed:

$ dpkg -l | grep linux-image-.*-generic | grep ^ii
ii linux-image-5.15.0-82-generic 5.15.0-82.91 amd64 Signed kernel image generic
ii linux-image-5.19.0-50-generic 5.19.0-50.50 amd64 Signed kernel image generic
ii linux-image-6.2.0-26-generic 6.2.0-26.26~22.04.1 amd64 Signed kernel image generic
ii linux-image-6.2.0-31-generic 6.2.0-31.31~22.04.1 amd64 Signed kernel image generic

I tried running sudo apt-get autoremove --purge but it doesn't work, due to the failed linux-firmware installation.

Should I file a new bug for this? I'm running Ubuntu 22.04 but that doesn't seem to be listed for this bug.

[bmaupin (bmaupin)]

So I did some more digging, and I guess the 5.15.0-82 kernel is there because it's required by linux-image-generic:

$ dpkg -l | grep linux-image | grep ^ii
ii linux-image-5.15.0-82-generic 5.15.0-82.91 amd64 Signed kernel image generic
ii linux-image-5.19.0-50-generic 5.19.0-50.50 amd64 Signed kernel image generic
ii linux-image-6.2.0-26-generic 6.2.0-26.26~22.04.1 amd64 Signed kernel image generic
ii linux-image-6.2.0-31-generic 6.2.0-31.31~22.04.1 amd64 Signed kernel image generic
ii linux-image-generic 5.15.0.82.78 amd64 Generic Linux kernel image
ii linux-image-generic-hwe-22.04 6.2.0.31.31~22.04.8

Although even after uninstalling linux-image-generic, sudo apt autoremove still doesn't remove any additional kernels. I guess somehow the logic that apt uses to remove unused kernels doesn't account for when multiple linux-image-generic packages are installed because of HWE.

[Julian Andres Klode (juliank)]

@bmaupin You'll want to use `apt list` to actually then also see in the output if they are all automatically installed, but if they all are feel free to open a new bug, there's not much point putting that in this one.

[Ivan T (ivanno)]

I had the same issue as bmaupin

/boot$ df -h .
Filesystem Size Used Avail Use% Mounted on
bpool/BOOT/ubuntu_azukaj 1.1G 590M 476M 56% /boot

/boot$ ls -lart
total 603325
drwxr-xr-x 4 root root 4096 1月 1 1970 efi
-rw-r--r-- 1 root root 184980 2月 7 2022 memtest86+_multiboot.bin
-rw-r--r-- 1 root root 184476 2月 7 2022 memtest86+.elf
-rw-r--r-- 1 root root 182800 2月 7 2022 memtest86+.bin
drwxr-xr-x 19 root root 25 10月 8 2022 ..
-rw------- 1 root root 6261790 5月 15 22:10 System.map-5.15.0-73-generic
-rw-r--r-- 1 root root 261914 5月 15 22:10 config-5.15.0-73-generic
-rw------- 1 root root 11574280 5月 15 22:50 vmlinuz-5.15.0-73-generic
-rw-r--r-- 1 root root 133982499 7月 13 13:28 initrd.img-5.15.0-73-generic
-rw------- 1 root root 6273612 8月 14 18:05 System.map-5.15.0-83-generic
-rw-r--r-- 1 root root 262053 8月 14 18:05 config-5.15.0-83-generic
-rw------- 1 root root 11615656 8月 14 18:07 vmlinuz-5.15.0-83-generic
-rw------- 1 root root 7967590 8月 16 18:42 System.map-6.2.0-31-generic
-rw-r--r-- 1 root root 275587 8月 16 18:42 config-6.2.0-31-generic
-rw------- 1 root root 13796616 8月 16 21:43 vmlinuz-6.2.0-31-generic
-rw------- 1 root root 7969006 8月 18 18:38 System.map-6.2.0-32-generic
-rw-r--r-- 1 root root 275587 8月 18 18:38 config-6.2.0-32-generic
-rw------- 1 root root 13791304 8月 18 18:40 vmlinuz-6.2.0-32-generic
-rw-r--r-- 1 root root 145066694 9月 3 06:34 initrd.img-6.2.0-31-generic
-rw-r--r-- 1 root root 132973394 9月 11 11:53 initrd.img-5.15.0-83-generic
lrwxrwxrwx 1 root root 25 9月 11 11:57 vmlinuz.old -> vmlinuz-5.15.0-83-generic
lrwxrwxrwx 1 root root 24 9月 11 11:57 vmlinuz -> vmlinuz-6.2.0-32-generic
lrwxrwxrwx 1 root root 28 9月 11 11:57 initrd.img.old -> initrd.img-5.15.0-83-generic
lrwxrwxrwx 1 root root 27 9月 11 11:57 initrd.img -> initrd.img-6.2.0-32-generic
-rw-r--r-- 1 root root 145040737 9月 11 11:58 initrd.img-6.2.0-32-generic
drwxr-xr-x 4 root root 27 9月 11 12:01 .
drwxr-xr-x 5 root root 4096 9月 21 11:53 grub

so I tried

/boot$ sudo apt purge linux-image-5.15.0-73-generic linux-headers-5.15.0-73-generic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  linux-image-unsigned-5.15.0-73-generic
Suggested packages:
  fdutils linux-doc | linux-source-5.15.0 linux-tools linux-headers-5.15.0-73-generic
The following packages will be REMOVED:
  linux-headers-5.15.0-73-generic* linux-image-5.15.0-73-generic*
The following NEW packages will be installed:
  linux-image-unsigned-5.15.0-73-generic
0 upgraded, 1 newly installed, 2 to remove and 213 not upgraded.
Need to get 11.7 MB of archives.
After this operation, 24.3 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 http://jp.archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-unsigned-5.15.0-73-generic amd64 5.15.0-73.80 [11.7 MB]
Fetched 11.7 MB in 5s (2,290 kB/s)
(Reading database ... 419503 files and directories currently installed.)
Removing linux-headers-...

[Julian Andres Klode (juliank)]

Just continue appending packages that APT would install until it stops trying to install new stuff.