various linux packages being marked as manually installed, still prevents 'apt-get autoremove' from doing the right thing for kernels

[Steve Langasek]

Note
----
This bug is referenced on https://help.ubuntu.com/community/RemoveOldKernels and in the event that this is fixed it'd be good to update the wiki page by likely removing the link to this bug.

On vivid, my system is full of kernels that have been tagged as manually installed again, not automatically installed; preventing 'apt-get autoremove' from doing the right thing.

$ sudo apt-mark auto linux-tools-3.19.0-{8,7,10,9}{,-generic} linux-headers-3.19.0-{10,7,8,9}{,-generic} linux-image{,-extra}-3.19.0-{10,7,8,9}-generic
linux-tools-3.19.0-8 set to automatically installed.
linux-tools-3.19.0-8-generic set to automatically installed.
linux-tools-3.19.0-7 set to automatically installed.
linux-tools-3.19.0-7-generic set to automatically installed.
linux-tools-3.19.0-10 was already set to automatically installed.
linux-tools-3.19.0-10-generic was already set to automatically installed.
linux-tools-3.19.0-9 set to automatically installed.
linux-tools-3.19.0-9-generic set to automatically installed.
linux-headers-3.19.0-10 was already set to automatically installed.
linux-headers-3.19.0-10-generic was already set to automatically installed.
linux-headers-3.19.0-7 set to automatically installed.
linux-headers-3.19.0-7-generic set to automatically installed.
linux-headers-3.19.0-8 set to automatically installed.
linux-headers-3.19.0-8-generic set to automatically installed.
linux-headers-3.19.0-9 set to automatically installed.
linux-headers-3.19.0-9-generic set to automatically installed.
linux-image-3.19.0-10-generic was already set to automatically installed.
linux-image-3.19.0-7-generic set to automatically installed.
linux-image-3.19.0-8-generic set to automatically installed.
linux-image-3.19.0-9-generic set to automatically installed.
linux-image-extra-3.19.0-10-generic was already set to automatically installed.
linux-image-extra-3.19.0-7-generic set to automatically installed.
linux-image-extra-3.19.0-8-generic set to automatically installed.
linux-image-extra-3.19.0-9-generic set to automatically installed.
$

The only reason -10 was marked as auto-installed was that it went via an 'apt-get dist-upgrade' instead of update-manager (because of the unrelated problem with debconf passthrough currently failing in vivid and blocking upgrades).

ProblemType: BugDistroRelease: Ubuntu 15.04
Package: update-manager 1:15.04.3
ProcVersionSignature: Ubuntu 3.19.0-7.7-generic 3.19.0
Uname: Linux 3.19.0-7-generic x86_64
ApportVersion: 2.17-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Thu Apr 2 07:58:41 2015
GsettingsChanges:
 b'com.ubuntu.update-manager' b'show-details' b'true'
 b'com.ubuntu.update-manager' b'window-height' b'716'
 b'com.ubuntu.update-manager' b'first-run' b'false'
 b'com.ubuntu.update-manager' b'window-width' b'729'
 b'com.ubuntu.update-manager' b'launch-time' b'1427986279'
InstallationDate: Installed on 2010-09-24 (1651 days ago)InstallationMedia: Ubuntu 10.04.1 LTS "Lucid Lynx" - Release amd64 (20100816.1)
PackageArchitecture: allSourcePackage: update-manager
UpgradeStatus: Upgraded to vivid on 2014-12-06 (117 days ago)

[Steve Langasek (vorlon)]

Michael, I know we've discussed before about how to possibly fix this, but it seems this has yet to be implemented. Can you please look at whether this can be implemented in time for 15.04? This continues to be a source of pain for our users, who receive 40MB of new kernel in /boot every 3 weeks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Benjamin Xiao (ben-r-xiao)]

Yes I am definitely seeing this behavior on Vivid with the latest kernel updates. Doing a regular apt-get dist-upgrade works fine and marks the new kernel packages as automatically installed. Using the update-manager always marks them as manually installed.

[Tyler Dinsmoor (pappad)]

The reasons to why old kernels are not intentionally being marked for autoremoval (as of 2008-11-04) is because it was considered too dangerous, and apt has an exception to not tag linux-image-* packages for autoremoval because _we_ had no idea which kernels actually worked for a system.

However, I had thought that this problem had been addressed using last-good-boot for grub after runlevel 2 was reached, here:

https://wiki.ubuntu.com/KernelTeam/removing-old-kernels

But since the wiki hasn't been updated since '08, maybe that is not true anymore.

[Steve Langasek (vorlon)]

On Tue, Jul 07, 2015 at 02:37:07PM -0000, Tyler Dinsmoor wrote:
> The reasons to why old kernels are not intentionally being marked for
> autoremoval (as of 2008-11-04) is because it was considered too
> dangerous,

No, absolutely not. The reason they're not being marked for autoremoval is
because there's a bug in the interface between update-manager and aptdaemon.

[SeijiSensei (ubuntuforums-2)]

I just ran dist-upgrade on my Kubuntu 14.04 system, and when it finished I had four kernels and associated files in /boot:
3.16.0.33-generic
3.16.0.38-generic
3.16.0.43-generic
3.16.0.44-generic

Autoremove failed to identify any of these. I had to run apt-get remove for the 33 and 38 kernels.

[Michael Vogt (mvo)]

The attached patches seems to work, I just tested the upgrade and get:
# apt list linux-image-extra-3.19.0-28-generic
Listing... Done
linux-image-extra-3.19.0-28-generic/vivid-proposed,now 3.19.0-28.30 amd64 [installed,automatic]

[Michael Vogt (mvo)]

I guess one could argue about a better API for the aptdaemon client for this, maybe "MarkAutoInstalled(pkg_list)" or Mark({Auto,Hold}, pkg_list".

[Jacob Nevins (0jacobnk-ulp)]

I just raised bug #1492709 about a similar but not obviously identical issue on Trusty (auto-marking only works if updates are set to automatically install). Is it the same issue?

[Steve Langasek (vorlon)]

Opening a bug task on the aptdaemon upstream project for this as well, since solving this involves making changes to the aptdaemon protocol. Sebastian, or other aptdaemon maintainers, do you have any thoughts here on mvo's proposed implementation of taking this optional 'auto' flag from clients for packages being installed?

[Steve Langasek (vorlon)]

Since this is a longstanding critical bug, in the absence of any response I'm going to go ahead with uploading both the aptdaemon and update-manager changes to wily, with an eye to SRUing it to trusty and vivid later.

This does break compatibility with older aptdaemon, so I am adding a versioned Depends from update-manager. If the aptdaemon patch is rejected upstream and needs to be reverted in Ubuntu (in favor of a different solution), then some versioned Breaks will be needed at that time.

[Launchpad Janitor (janitor)]

This bug was fixed in the package aptdaemon - 1.1.1+bzr982-0ubuntu13

---------------
aptdaemon (1.1.1+bzr982-0ubuntu13) wily; urgency=medium

  * Drop Vcs-Bzr from debian/control:
    - The branch pointed to is "ubuntu-vivid". This is obviously incorrect
      for wily; and you shouldn't have to do a busywork update of Vcs-Bzr
      with each release.
    - Uploaders do not have commit rights on the listed branch. A Vcs-Bzr
      field should only point to a branch that uploaders have rights to,
      otherwise it can never be authoritative.
  * debian/patches/lp1439769-aptdaemon-autoinstall.patch: Take a flag to
    indicate whether a requested package is auto-installed. Thanks to
    Michael Vogt <email address hidden>. Closes LP: #1439769.

 -- Steve Langasek <email address hidden> Tue, 06 Oct 2015 11:33:41 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:15.10.3

---------------
update-manager (1:15.10.3) wily; urgency=medium

  * Pass a '#auto' suffix to aptdaemon for packages that are autoinstalled,
    so that we have enough information to autoremove them later. Thanks to
    Michael Vogt <email address hidden> for the patch. LP: #1439769.
  * Add versioned dependency on aptdaemon (>= 1.1.1+bzr982-0ubuntu13) for the
    above.

 -- Steve Langasek <email address hidden> Tue, 06 Oct 2015 21:26:35 -0700

[Jarno Suni (jarnos)]

Steve Langasek, there is still no fix released for Trusty (and maybe Precise?). Bug #1492709

[Jarno Suni (jarnos)]

See also Bug #1175637

[Jarno Suni (jarnos)]

SRU for 14.04 maybe?

[Jarno Suni (jarnos)]

...so that unattended-upgrades / 'sudo apt-get autoremove' could remove extra kernels installed by Software Updater.

[Jarno Suni (jarnos)]

... that would be a great advantage, if /boot is about full, allowing installation of further kernel updates.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Balint Reczey (rbalint)]

This is fixed in Xenial and later in LP: #1675079