Kernel updates marked as "manually installed", prevents old kernels cleanup

[Erich Eickmeyer]

With the machines I help administer, we have been finding situations where the /boot directory is filling-up beyond 3 kernels on LUKS encrypted systems. apt autoremove is not removing old kernels as expected.

This may also be an issue with unattended-upgrades since I found the following line commented-out by default:

// Remove unused automatically installed kernel-related packages
// (kernel images, kernel headers and kernel version locked tools).
// Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";

We have had a system with as many as 15 kernel packages installed as a result of this not working as expected.

The majority of these machines are using Discover to do their package upgrading, which uses PackageKit as its backend.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: apt 2.0.4
ProcVersionSignature: Ubuntu 5.8.0-41.46~20.04.1-generic 5.8.18
Uname: Linux 5.8.0-41-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.14
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: KDE
Date: Tue Feb 2 09:29:01 2021
InstallationDate: Installed on 2020-11-07 (87 days ago)
InstallationMedia: Kubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
SourcePackage: apt
UpgradeStatus: No upgrade log present (probably fresh install)

[Balint Reczey (rbalint)]

The configuration file is fine, it is the default value that's commented out.

Please check unattended-upgrades logs if it performed kernel removals:
zgrep 'unused kernel' /var/log/unattended-upgrades/*.gz

Also please check if it ran at all.

[Erich Eickmeyer (eeickmeyer)]

Hi Balint!

Unfortunately, it looks like it didn't run at all. On my system, I came up with this:

root@erich-KFocus:/var/log/unattended-upgrades# grep 'unused kernel' /var/log/unattended-upgrades/*.gz
root@erich-KFocus:/var/log/unattended-upgrades#

That tells me it likely didn't run at all.

[Balint Reczey (rbalint)]

Is unatteded-upgrade disabled on the system?
$ cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

$ apt-config dump | grep Periodic
APT::Periodic "";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "1";

[TJ (tj)]

I'm also wondering if these are laptops or desktops, and if laptops, are they mainly on battery whilst not sleeping/charging? I ask since apt-daily-upgrade.service has:

ConditionACPower=true

and so may be reporting:

Condition check resulted in Daily apt upgrade and clean activities being skipped.

[Erich Eickmeyer (eeickmeyer)]

Balint:

Here's the results, looks like everything matches what you've got there.

erich@erich-KFocus:~$ cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "1";
erich@erich-KFocus:~$ apt-config dump | grep Periodic
APT::Periodic "";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "1";
erich@erich-KFocus:~$

TJ: These are high-performance laptop computers and are usually plugged-in. Our clients include NASA's Jet Propulsion Laboratory.

[Balint Reczey (rbalint)]

@eeickmeyer Please run manually:

sudo unattended-upgrade --verbose --debug

[Erich Eickmeyer (eeickmeyer)]

Balint:

Here's the result:
https://paste.ubuntu.com/p/9kbT6HDpjw/

Bear in mind, I have a few external repositories, and we also ship a bunch of external repositories by default. That said, none of those should affect this as they don't contain kernels, nor do we pin any kernels.

If you'd like, I'll ask the company president to give me the output of the same command.

[TJ (tj)]

I haven't been able to test this since the systems I have here don't have any kernel images lying around, but wondering if this patch could help narrow down the cause?

Apply to a (copy) of /usr/bin/unattended-upgrade - possibly put in /usr/local/bin/ so it is called before the package-installed version (don't forget to clear the bash hash cache ("hash -r") first)

Then:

sudo /usr/local/bin/unattended-upgrade --verbose --debug

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Kernel pkg debugging" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Erich Eickmeyer (eeickmeyer)]

Hi TJ,

Here's the output from the patched version:
https://paste.ubuntu.com/p/dzGqfR9PRH/

I hope this helps. Thanks for chipping-in!

[TJ (tj)]

Erich, looks like that code wasn't reached or if it was the object (list) "auto_removable" was empty.

Maybe the problem is outside this code. Does apt think those packages are still installed?

apt list --installed 'linux-image*'

Also, what is the actual content of /boot/ ("ls -latr /boot/") ? I'm wondering if it could be that the kernels (linux-image-*) packages have been removed but the (space-eating) initrd,img* files have remained. Each of those can be close to 50MB.

[Balint Reczey (rbalint)]

Please install a few old kernels manually, mark them autoremovable and run u-u in verbose and debug mode.

If u-u removes those kernels this is not an u-u bug.

Then install those again and try removing them with apt.

If apt removes those kernels than this is not an APT bug.

[Erich Eickmeyer (eeickmeyer)]

TJ:

linux-image-5.4.0-40-generic/focal-updates,focal-security,now 5.4.0-40.44 amd64 [installed,automatic]
linux-image-5.4.0-40-lowlatency/focal-updates,focal-security,now 5.4.0-40.44 amd64 [installed,automatic]
linux-image-5.4.0-42-generic/focal-updates,focal-security,now 5.4.0-42.46 amd64 [installed,auto-removable]
linux-image-5.4.0-42-lowlatency/focal-updates,focal-security,now 5.4.0-42.46 amd64 [installed,auto-removable]
linux-image-5.4.0-65-generic/focal-updates,focal-security,now 5.4.0-65.73 amd64 [installed]
linux-image-5.4.0-65-lowlatency/focal-updates,focal-security,now 5.4.0-65.73 amd64 [installed]
linux-image-5.8.0-34-generic/focal-updates,focal-security,now 5.8.0-34.37~20.04.2 amd64 [installed,automatic]
linux-image-5.8.0-34-lowlatency/focal-updates,focal-security,now 5.8.0-34.37~20.04.2 amd64 [installed,automatic]
linux-image-5.8.0-41-generic/focal-updates,focal-security,now 5.8.0-41.46~20.04.1 amd64 [installed]
linux-image-5.8.0-41-lowlatency/focal-updates,focal-security,now 5.8.0-41.46~20.04.1 amd64 [installed]
linux-image-generic-hwe-20.04/focal-updates,focal-security,now 5.8.0.41.46~20.04.27 amd64 [installed,automatic]
linux-image-generic/focal-updates,focal-security,now 5.4.0.65.68 amd64 [installed,automatic]
linux-image-lowlatency-hwe-20.04/focal-updates,focal-security,now 5.8.0.41.46~20.04.27 amd64 [installed,automatic]
linux-image-lowlatency/focal-updates,focal-security,now 5.4.0.65.68 amd64 [installed,automatic]

Balint:
Result: https://paste.ubuntu.com/p/JVHZpFWBBT/

It appears as though apt and u-u are doing the job, but for some reason the packages aren't getting marked as autoremovable when they're upgraded? I had to manually mark these as autoremovable (apt-mark linux-image-${version}-${generic|lowlatency} auto) to get u-u or apt to do anything about it. If you look at the list above, even the 5.4.0-65 and 5.4.0-41 kernels are showing up as manually installed, which is not the case (they were installed via their respective metapackages).

So, something is definitely not working as expected.

[Balint Reczey (rbalint)]

APT and u-u and marking work as designed.

[Erich Eickmeyer (eeickmeyer)]

Since we narrowed it down, I filed the following upstream bug: https://bugs.kde.org/show_bug.cgi?id=432513

[Aex Rauch (centaurie)]

this bug is somehow in packagekit. I do not use some ui/gui for packagekit, nor unattended-upgrades nor ather automatic update tools. Normally I use apt.

apt-mark showmanual | egrep linux-'[g|m|h|i]'
linux-generic
linux-headers-5.4.0-73
linux-headers-5.4.0-73-generic
linux-image-5.4.0-73-generic
linux-modules-5.4.0-73-generic
linux-modules-extra-5.4.0-73-generic

the kernel comes new.
This appears after I upgrade with pkcon from commandline. As long I use apt all is okay.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Erich Eickmeyer (eeickmeyer)]

Apparently this is a known issue with PackageKit, but hasn't been addressed by the developers. https://github.com/PackageKit/PackageKit/issues/450

[Mauro (mauromol)]

See also: https://bugs.launchpad.net/ubuntu/+source/plasma-discover/+bug/1914932

[Erich Eickmeyer (eeickmeyer)]

Attached is a .debdiff of PackageKit 1.2.4 in the archive compared with the latest git snapshot with this bugfix. The PackageKit team is prepping a release of 1.2.5. While the attached .debdiff represents a snapshot of the git repository on 2022-02-15, it is very close to what the release of 1.2.5 will become. For this reason, it should be safe and 1.2.5 will include very minute changes by comparison.

[Julian Andres Klode (juliank)]

1.2.5 is out now, I'll go grab that directly rather than try to reconstruct the correct ugit tarball for the git snapshot :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package packagekit - 1.2.5-1ubuntu1

---------------
packagekit (1.2.5-1ubuntu1) jammy; urgency=low

  * Merge from Debian unstable. Remaining changes:
    - debian/control: Added recommends for packagekit-installer
  * 1.2.5 fixes manually installed kernel updates (LP: #1914278)

packagekit (1.2.5-1) unstable; urgency=medium

  * New upstream version: 1.2.5
    Notable changes:
    - aptcc: Do not mark all updates as manually installed
    - aptcc: Apply auto/manual state from previously resolved pkgids
      when needed
    - aptcc: Attempt to fix broken packages again when system
      repair is requested
    - aptcc: Correctly list all the available package versions
    - aptcc: Display an excerpt of dpkg's log on error instead of not
      failing at all
    - aptcc: Emit only the first, likely highest-quality error when dpkg fails
    - aptcc: Make package downgrades work again
    - aptcc: Return local file listings in the same format as installed ones
    - Add flags to D-Bus offline invoking methods
    - Properly handle allow-reinstall flag for installations
    - Provide better error message if trying to install an installed package
    - Wait until online to activate systemd service
  * policy: Allow any inactive user to refresh the package cache
    as well (Closes: #965030)
  * Update symbols file
  * Mark d/rules as not requiring root
  * Bump dh compat level to 13
  * Drop some no longer needed breaks constraints
  * policy: Allow any user to trigger offline updates
    by default (Closes: #1003141)
  * Remove 2 obsolete maintscript entries in 1 files (thanks, Janitor!)
  * Set upstream metadata fields: Bug-Database, Contact (thanks, Janitor!)

 -- Julian Andres Klode <email address hidden> Mon, 21 Feb 2022 10:40:12 +0100

[Mauro (mauromol)]

Will this be backported to Focal?

[Matthias Klumpp (ximion)]

Resolving this required a fair bit of refactoring, so TBH I am not sure that it is safe to backport to Focal, unless version 1.2.5 is submitted as-a-whole.

[Mauro (mauromol)]

I see, just please consider that this problem is quite important for system stability after kernel upgrades (even security ones) and 20.04 is still the current LTS...

[Michael Mikowski (kfocus)]

@ximion thanks for this substantial work. The list of enhancements looks impressive. In testing on Jammy where I compare this to the prior versions, manual and automatic marks are as before which is good. We haven't confirmed meta package upgrades that pull in new dependencies (as with kernel packages) because we haven't seen an upgrade yet. However, we may build a package to test this.

[Matthias Klumpp (ximion)]

I tested this with a bunch of dummy packages and a local repository, and everything worked pretty well and states were applied as expected. Of course, doesn't mean that there may be a real-world bug that I haven't seen yet, but I think it's unlikely (I've also been running this on my machine on Debian for the past weeks).
Even automatic kernel removals work, but I could not test yet whether they also work through PackageKit.
Is Focal using PackageKit for updates even? I thought it was still on update-manager/aptd for that task...

[Erich Eickmeyer (eeickmeyer)]

Hi Matthias. To clarify, Kubuntu and Ubuntu Studio (from 20.10) use Discover for updates, which, in turn, uses PackageKit for updates.

[Julian Andres Klode (juliank)]

We'll continue/start working on the SRUs mid May, if no regressions are reported until then. That gives it a month or so in stable jammy which seems reasonable for the scope of changes and very little testing being done so far.

[Matthias Klumpp (ximion)]

Makes sense to me :-)

[Erich Eickmeyer (eeickmeyer)]

Hi Julian,

Sounds good, but there's one caveat: If the Kubuntu users receive a packagekit upgrade via Discover, packagekit will attempt to upgrade itself while running. This causes a packagekit crash, from which the recovery is via the command line:

sudo dpkg --configure -a
sudo apt --fix-broken install
sudo apt full-upgrade

At this point, everything returns to normal, but for many users this can be problematic, especially in a stable release. While not a regression per se, the process would be problematic for Kubuntu users as they're the ones in Focal using this upgrade method. More advanced users can navigate this, but non-technical users would have a rough time.

[Matthias Klumpp (ximion)]

> packagekit will attempt to upgrade itself while running. This causes a packagekit crash

That should not happen - do you have a backtrace for that? Reporting it at the PackageKit bugtracker would also be nice so I have it on my radar.

[Erich Eickmeyer (eeickmeyer)]

I was unable to reproduce the crash in a VM, so it must've been unrelated to packagekit. Therefore, I guess we're OK, but I'm keeping an eye on it.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.