Unattended upgrades removed linux-image-generic

[Dean Henrichsmeyer]

[Impact]
If a user accidentally removed linux-generic, unattended-upgrades will also autoremove linux-image-generic, leaving them without kernel upgrades.

[Test Case]
1. Remove all reverse dependencies of linux-image-generic
2. Mark linux-image-generic as automatically installed
3. Run unattended-upgrades -v --dry-run --debug to ensure that linux-image-generic is not removed (after verifying that it is, with old apt)

[Regression potential]
This adds two regular expressions to the list of packages that must not be automatically removed. As such, the only possible regression is that some packages starting with linux-image, not containing any dots, are not removed

[Original bug report]
On a fairly fresh install of 18.04 with no modifications whatsoever to the unattended-upgrades configuration, it decided to remove linux-image-generic which also removed linux-modules-extra which caused sound drivers to disappear, etc.

The relative snippet from /var/log/unattended-upgrades/unattended-upgrades.log is:

2018-08-15 06:18:00,048 INFO Starting unattended upgrades script
2018-08-15 06:18:00,048 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2018-08-15 06:18:01,552 INFO Removing unused kernel packages: linux-headers-generic linux-image-generic linux-headers-4.15.0-32-generic linux-headers-4.15.0-32
2018-08-15 06:18:01,588 WARNING Keeping auto-removable linux-headers-generic package(s) because it would also remove the following packages which should be kept in this step: libxml2 linux-image-4.15.0-32-generic linux-modules-4.15.0-32-generic linux-modules-extra-4.15.0-32-generic
2018-08-15 06:18:09,476 INFO Packages that were successfully auto-removed: linux-headers-4.15.0-32 linux-headers-4.15.0-32-generic linux-headers-generic linux-image-generic
2018-08-15 06:18:09,477 INFO Packages that are kept back: linux-headers-generic
2018-08-15 06:18:10,300 INFO Packages that will be upgraded: libxml2 linux-image-generic
2018-08-15 06:18:10,300 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2018-08-15 06:18:39,238 INFO All upgrades installed
2018-08-15 06:18:42,818 INFO Packages that were successfully auto-removed: linux-image-generic linux-modules-extra-4.15.0-32-generic
2018-08-15 06:18:42,818 INFO Packages that are kept back:

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: unattended-upgrades 1.1ubuntu1.18.04.5
ProcVersionSignature: Ubuntu 4.15.0-32.35-generic 4.15.18
Uname: Linux 4.15.0-32-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.3
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Thu Aug 16 13:17:30 2018
InstallationDate: Installed on 2018-07-24 (23 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180724)
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: unattended-upgrades
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Murray (brian-murray)]

Looking at .var.apt.history.log.txt we can see that in a previous package operation linux-generic was removed.

Start-Date: 2018-08-14 13:37:10
Commandline: apt dist-upgrade
Install: linux-headers-4.15.0-32-generic:amd64 (4.15.0-32.35, automatic), linux-headers-4.15.0-32:amd64 (4.15.0-32.35, automatic)
Upgrade: linux-headers-generic:amd64 (4.15.0.31.33, 4.15.0.32.34), linux-libc-dev:amd64 (4.15.0-31.33, 4.15.0-32.35)
Remove: linux-signed-generic:amd64 (4.15.0.31.33), linux-generic:amd64 (4.15.0.31.33)
End-Date: 2018-08-14 13:37:16

Subsequently during the unattended-upgrades run linux-image-generic is identified as an unused package. I'm only able to recreate this if I first remove the linux-generic package e.g.:

bdmurray@clean-bionic-amd64:~$ sudo apt remove linux-generic
[sudo] password for bdmurray:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  amd64-microcode intel-microcode iucode-tool linux-headers-generic linux-image-generic thermald
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  linux-generic
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 15.4 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 165999 files and directories currently installed.)
Removing linux-generic (4.15.0.30.32) ...
bdmurray@clean-bionic-amd64:~$ sudo unattended-upgrades -v
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
Removing unused kernel packages: linux-headers-generic linux-image-generic
(Reading database ... 165996 files and directories currently installed.)
Removing linux-headers-generic (4.15.0.30.32) ...
(Reading database ... 165993 files and directories currently installed.)
Removing linux-image-generic (4.15.0.30.32) ...
Packages that were successfully auto-removed: linux-headers-generic linux-image-generic
Packages that are kept back:
No packages found that can be upgraded unattended and no pending auto-removals

[Brian Murray (brian-murray)]

Looking some more at the apt history log file we can see mutter version 3.28.3-2~ubuntu18.04.1 being installed. This package only exists in the -proposed version of the repository.

[Adam Conrad (adconrad)]

So, this might partially be a linux-meta/overrides bug. The top-level metas (linux-$flavour) should be in the metapackages section so that if they're removed (for instance, by someone who doesn't want headers, but kinda likes kernels), the manually installed bit is transferred down one level to linux-image-$flavour and linux-headers-$flavour.

It might also be valid to say that the neverautoremove apt/kernel magic could/should include the mid-level metas in the list, so they're just considered manual regardless of what state they were accidentally put in.

[Adam Conrad (adconrad)]

It does appear, however, that despite the above comment (which I think highlights ways we can make this foot a smaller target), this definitely began with human error:

Start-Date: 2018-08-14 13:37:10
Commandline: apt dist-upgrade
Install: linux-headers-4.15.0-32-generic:amd64 (4.15.0-32.35, automatic), linux-headers-4.15.0-32:amd64 (4.15.0-32.35, automatic)
Upgrade: linux-headers-generic:amd64 (4.15.0.31.33, 4.15.0.32.34), linux-libc-dev:amd64 (4.15.0-31.33, 4.15.0-32.35)
Remove: linux-signed-generic:amd64 (4.15.0.31.33), linux-generic:amd64 (4.15.0.31.33)
End-Date: 2018-08-14 13:37:16

That's a manual apt dist-upgrade which would have asked for confirmation on that removal. unattended-upgrades alone would have never gotten into this situation, as it won't remove to force an upgrade.

[Julian Andres Klode (juliank)]

As Adam mentioned, the top-level kernel meta packages should change their section to metapackages, so we don't mark linux-image-generic for autoremoval if linux-generic is removed.

I'm looking into adding additional NeverAutoRemove entries for the metapackages to apt as

 "^linux-image-[a-z0-9]*$";
 "^linux-image-[a-z0-9]*-[a-z0-9]*$";

but I'm not sure if that catches all meta packages or more than it should.

[David Kalnischkies (donkult)]

They both sound awfully "generic" (pun intended), but I can't really come up with an example for a bad match, so lets just try it I guess.

That said, isn't the "deeper" problem that these metapackages can be removed easily even through they are important to keep the kernel up-to-date (and hence secure). Perhaps some should be "Important:yes"…

For the record: apts behavior is different depending on if you explicitly remove a metapackage vs. its removed as part of problem resolution like in a dist-upgrade or if you remove some dependee. Only in the later cases the manual bit is applied to the (other) dependencies of the metapackage.

[Julian Andres Klode (juliank)]

apt-side fix in

 https://salsa.debian.org/apt-team/apt/commit/a4b0ce5a4f5068f780b3aa94473230b5093a837d

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.8.0~alpha2ubuntu1

---------------
apt (1.8.0~alpha2ubuntu1) disco; urgency=medium

  * Adjust libapt-pkg Breaks aptitude to << 0.8.9

 -- Julian Andres Klode <email address hidden> Wed, 14 Nov 2018 12:07:59 +0100

[Brian Murray (brian-murray)]

Hello Dean, or anyone else affected,

Accepted apt into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Dean, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Dean, or anyone else affected,

Accepted apt into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.7.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Dean, or anyone else affected,

Accepted apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.6.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

cosmic:

before (1.7.0ubu...)
Removing unused kernel packages: linux-headers-generic linux-image-generic
after (1.7.2):
Removing unused kernel packages: linux-headers-generic

That does the job well enough, even if the headers end up being removed in this particular test case.

[Julian Andres Klode (juliank)]

(in practice the headers do not matter - if you have dkms installed, it will keep the headers alive)

[Julian Andres Klode (juliank)]

bionic:

before (1.6.6..)
Removing unused kernel packages: linux-headers-generic linux-image-generic
after (1.6.8):
Removing unused kernel packages: linux-headers-generic

and just to verify, if you install dkms:
No packages found that can be upgraded unattended and no pending auto-removals

(the headers won't get removed then)

[Łukasz Zemczak (sil2100)]

As per our discussion, releasing this before the 7-day aging period.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.7.2

---------------
apt (1.7.2) cosmic; urgency=medium

  * Merge security update content injection in http method (CVE-2019-3462)

apt (1.7.1) cosmic; urgency=medium

  * Add support for /etc/apt/auth.conf.d/*.conf (netrcparts) (LP: #1811120)
  * NeverAutoRemove kernel meta packages (LP: #1787460)
  * Merge translations from 1.8 series
  * Source-only changes:
    - debian/gbp.conf: Point to 1.7.y branch
    - Do CI using ubuntu:cosmic, not debian:testing

 -- Julian Andres Klode <email address hidden> Fri, 25 Jan 2019 12:41:42 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.6.8

---------------
apt (1.6.8) bionic; urgency=medium

  * merge security update: content injection in http method (CVE-2019-3462)

apt (1.6.7) bionic; urgency=medium

  [ Milo Casagrande ]
  * [l10n] Update Italian translation

  [ Julian Andres Klode ]
  * NeverAutoRemove kernel meta packages (LP: #1787460)
  * Add support for /etc/apt/auth.conf.d/*.conf (netrcparts) (LP: #1811120)
  * Merge translations from 1.8 series

 -- Julian Andres Klode <email address hidden> Fri, 25 Jan 2019 12:51:00 +0100

[Brian Murray (brian-murray)]

Hello Dean, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.30 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Dean, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Dean, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.31 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Dean, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

So, unattended-upgrades refused to treat linux-image-generic as autoremovable in xenial and trusty; so let's look at apt itself:

In both cases (1.2.31, and 1.0.1ubuntu2.22), linux-image-generic is not autoremovable after removing linux-generic, whereas it was in released updates - fix verified.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.2.31

---------------
apt (1.2.31) xenial; urgency=medium

  * Fix name of APT::Update::Post-Invoke-Stats (was ...Update-Post...)
  * apt.dirs: Install auth.conf.d directory (LP: #1818996)
  * Merge translations from 1.6.10 (via 1.4.y branch)

apt (1.2.30) xenial; urgency=medium

  * merge security upload for content injection in http method (CVE-2019-3462);
    with fixed autopkgtest (LP: #1815750)
  * Introduce experimental 'never' pinning for sources (LP: #1814727)
  * Add support for /etc/apt/auth.conf.d/*.conf (netrcparts) (LP: #1811120)
  * Add a Packages-Require-Authorization Release file field (LP: #1814727)
  * NeverAutoRemove kernel meta packages (LP: #1787460)
  * doc: Set ubuntu-codename to xenial (LP: #1812696)
  * update: Provide APT::Update-Post-Invoke-Stats script hook point
    (LP: #1815760)
  * Introduce APT::Install::Pre-Invoke / Post-Invoke-Success (LP: #1815761)

 -- Julian Andres Klode <email address hidden> Tue, 12 Mar 2019 14:59:01 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.1ubuntu2.22

---------------
apt (1.0.1ubuntu2.22) trusty; urgency=medium

  * apt.dirs: Install auth.conf.d directory (LP: #1818996)
  * Merge translations from 1.2.31

apt (1.0.1ubuntu2.21) trusty; urgency=medium

  [ Julian Andres Klode ]
  * travis CI: Use docker container to get useful results
  * fix and non-silent fail dpkg-overwrite error test (LP: #1817088)
  * Introduce experimental 'never' pinning for sources (LP: #1814727)
  * Add support for /etc/apt/auth.conf.d/*.conf (netrcparts) (LP: #1811120)
  * Add a Packages-Require-Authorization Release file field (LP: #1814727)
  * NeverAutoRemove kernel meta packages (LP: #1787460)
  * Introduce APT::Install::Pre-Invoke / Post-Invoke-Success (LP: #1815761)

  [ David Kalnischkies ]
  * ftparchive/writer.cc: use a std::vector instead of hardcoded array
    (LP: #1817048)

 -- Julian Andres Klode <email address hidden> Tue, 12 Mar 2019 15:15:54 +0100

[Mathew Hodson (mhodson)]

Are we still going to change the section of the metapackages? Those tasks are still open.

[m4dm4x1337 (m4dm4x1337)]

Hi,

this bug still exists! It is IMPOSSIBLE to install linux-generic-hwe-18.04 and virtualbox virtualbox-guest-x11-hwe virtualbox-ext-pack at the same time. (arch i386)

I already reported this problem month ago here:
https://bugs.launchpad.net/ubuntu/+source/linux-meta-hwe/+bug/1856492

Seems like nobody cares! This is just pure shit! Some idiots think they have to release kernel 5.3, but they don't consider that nothing will work afterwards. What is even more stupid is that Kernel 5.0 is no longer available! idiots! idiots! idiots! Damn, fix that!

How to reproduce this bug?
--------------------------

1.) create a minimal ubuntu with debootstrap

2.) use this sources.list:

deb http://de.archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu bionic-proposed main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse
deb http://archive.canonical.com/ubuntu bionic partner

3.) update package lists

apt-get update

4.) install kernel 5.3.0 from bionic-proposed

apt-get install linux-generic-hwe-18.04

5.) install virtualbox

apt-get install virtualbox virtualbox-guest-x11-hwe virtualbox-ext-pack

---------------------------------------

actual result:

linux-generic-hwe-18.04 is being removed due to invalid dependencies

expected result:

linux-generic-hwe-18.04 keeps installed

[Steve Langasek (vorlon)]

Ubuntu 19.04 reached end of life on January 23, 2020. I am therefore marking this bug 'wontfix' for that release.

[Mathew Hodson (mhodson)]

The issues with virtualbox and hwe can be tracked in bug 1866518 or bug 1874309