By default settings unattended-upgrade does not automatically remove packages that become unused in conjunction with updating by other software

Bug #1624644 reported by Jarno Suni on 2016-09-17
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Undecided
Unassigned
Artful
Undecided
Unassigned
gnome-software (Ubuntu)
Undecided
Unassigned
Artful
Undecided
Unassigned
unattended-upgrades (Ubuntu)
High
Balint Reczey
Artful
High
Balint Reczey
update-manager (Ubuntu)
Undecided
Unassigned
Artful
Undecided
Unassigned

Bug Description

When using default settings for unattended-upgrade i.e.
Unattended-Upgrade::Remove-Unused-Dependencies "false";
# default "false"
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
# default "true"
in configuration file /etc/apt/apt.conf.d/50unattended-upgrades,
unattended-upgrade is unable to remove packages that become unused in conjunction with updating by other software such as update-manager or apt full-upgrade. This is because unattended-upgrade compares the list of unneeded packages before and after it upgrades packages to detect which packages are new unused ones.

Consequently, if user installs new kernels using e.g. update-manager, the excessive kernels will not be removed by unattended-upgrade, and eventually (small) /boot will become full.

Expected behavior: handle removing of unused packages differently at least until other package management software installed by default can handle removing of new unused packages.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: unattended-upgrades 0.90
ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16
Uname: Linux 4.4.0-36-generic i686
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: i386
CurrentDesktop: XFCE
Date: Sat Sep 17 11:28:44 2016
InstallationDate: Installed on 2016-09-05 (11 days ago)
InstallationMedia: Mythbuntu 16.04.1 LTS "Xenial Xerus" - Release i386 (20160719)
PackageArchitecture: all
SourcePackage: unattended-upgrades
UpgradeStatus: No upgrade log present (probably fresh install)

Jarno Suni (jarnos) wrote :
description: updated
Ian Weisser (ian-weisser) wrote :

Can you please talk us through exactly how to reproduce the issue?
I have used u-u with a wide variety of other apt frontends in 16.04 without seeing that problem yet.

Jarno Suni (jarnos) wrote :

Install each kernel update by update-manager and notice no kernels will be automatically removed by U-U.

Jarno Suni (jarnos) on 2016-09-21
description: updated
summary: - Unable to remove packages that become unused in conjunction with
- updating by other software
+ Unable to automatically remove packages that become unused in
+ conjunction with updating by other software
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed

I find that on most times the bug occurs, but not always. Though it has caused me some problems before with /boot filling up and me being forced to manually remove the old kernels.

Jarno Suni (jarnos) wrote :

Nikita, what do you mean by that it does not occur always?

Jarno Suni (jarnos) wrote :

I run /etc/kernel/postinst.d/apt-auto-removal on each boot. That also makes the "Unattended-Upgrade::Remove-New-Unused-Dependencies" setting not work.
(See Bug #1615381.)

Changed in unattended-upgrades (Ubuntu):
importance: Undecided → High
tags: added: rls-z-incoming
Jarno Suni (jarnos) wrote :

A fix could be adding line
Unattended-Upgrade::Remove-Unused-Dependencies "true";
in /etc/apt/apt.conf.d/50unattended-upgrades
But does it remove too many packages? Why was
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
setting ever added?

Jarno Suni (jarnos) wrote :

Besides:
If you do not want automatic updates, but just automatic removal of unnecessary software including kernels, you could have 'Download and install updates automatically' set, but edit /etc/apt/apt.conf.d/50unattended-upgrades to have only comments within Unattended-Upgrade::Allowed-Origins section.

tags: added: rls-aa-incoming
removed: rls-z-incoming
tags: removed: rls-aa-incoming
Steve Langasek (vorlon) on 2017-06-22
Changed in unattended-upgrades (Ubuntu Artful):
assignee: nobody → Steve Langasek (vorlon)
assignee: Steve Langasek (vorlon) → Canonical Foundations Team (canonical-foundations)
Changed in unattended-upgrades (Ubuntu Artful):
assignee: Canonical Foundations Team (canonical-foundations) → Balint Reczey (rbalint)
Balint Reczey (rbalint) wrote :

@Jarno: IMO Unattended-Upgrade::Remove-Unused-Dependencies is already a risky option and I don't recommend enabling it because it may remove packages which are not used according the to package-dependency chain but which users rely on using software that is not packaged.
The only place I would use this option are dedicated server systems where everything is packaged and which run services only without allowing user shell/GUI logins.

This bug is about removing packages which became unused as a result of running other package management tools, not u-u.

I consider this out of scope for u-u because it can be solved easily by the used other package management tool and this feature will most likely unpleasantly surprise users by removing unexpected packages.

Changed in unattended-upgrades (Ubuntu Artful):
status: Confirmed → Opinion
Balint Reczey (rbalint) wrote :

I suggest marking that bug as Won't Fix.

Balint Reczey (rbalint) wrote :

Added update-manager as affected package because update-manager is the tool leaving newly unused packages around.

Jarno Suni (jarnos) wrote :

rbalint, user could mark such packages as manually installed (by apt-mark), so autoremove will not remove them.

If you mark update-manager affected, you should mark other updaters, too. E.g. 'apt full-upgrade' or Synaptic will not remove newly unused packages, right?

Jarno Suni (jarnos) wrote :

I think less advanced users do not install software that is not packaged. Advanced users can configure system to work as they wish. IMO currently Unattended-Upgrade::Remove-New-Unused-Dependencies is more risky as default.

Balint Reczey (rbalint) wrote :

@Jarno I added update-manager based on my experience with (less experienced) users, who kept their system up-to-date by saying yes to everything regarding updates which popped up on their system.

They never touched apt or synaptic nor installed packages by themselves by other means.

I think removing newly unused packages would be a reasonable default for u-u and update-manager.

I agree that Remove-New-Unused-Dependencies=true is riskier than having it disabled but it still removes a subset of what Remove-Unused-Dependencies=true would thus Remove-Unused-Dependencies=true would be more risky.

Remove-New-Unused-Dependencies=true is needed to clean up old kernel packages filling up /boot (and the whole disk).
If update-manager defaults to that by default, too, the system won't break itself.

Jarno Suni (jarnos) wrote :

I think removing "newly unused" packages is an ugly hack. Deciding which packages are excessive or unneeded should not depend on how some packages were removed previously. If the reason for this automatic remove thing is to get rid of excessive kernels, a specific tool for that could be used. I have written such a tool. It is called linux-purge (https://launchpad.net/linux-purge)

Jarno Suni (jarnos) wrote :

I meant deciding which packages are excessive or unneeded should be possible also after upgrading, not only in conjunction with upgrading.

I still do not get it why advanced users could not make sure the packages needed by their not packaged software is not marked as manually installed.

Jarno Suni (jarnos) wrote :

Minus last "not"

Jarno Suni (jarnos) wrote :

Anyway, it would be good, if other software could optionally remove new unused packages.
I do not consider Unattended-Upgrade::Remove-New-Unused-Dependencies as an reasonable default option for unattended-upgrades before the feature has been released for at least apt, update-manager, gnome-software and possible other related software installed by default.

Jarno Suni (jarnos) on 2017-09-18
summary: - Unable to automatically remove packages that become unused in
- conjunction with updating by other software
+ By default settings unattended-upgrade is unable to automatically remove
+ packages that become unused in conjunction with updating by other
+ software.
description: updated
Jarno Suni (jarnos) on 2017-09-18
description: updated
Changed in unattended-upgrades (Ubuntu Artful):
status: Opinion → New
Jarno Suni (jarnos) on 2017-09-20
tags: added: full-boot
Brian Murray (brian-murray) wrote :

I found the description somewhat confusing because 'Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; doesn't actually exist in /etc/apt/apt.conf.d/50unattended-upgrades, rather that's the default setting in unattended-upgrades since 0.90. Regardless, I was able to recreate the issue on an Ubuntu 16.04 system via the following procedure.

1) Use apt-get install to install a new kernel (linux-image-generic)
2) Run unattended-upgrades (observe "no pending auto-removals")

However, running 'sudo apt autoremove' does want to remove an old kernel.

Jarno Suni (jarnos) wrote :

Brian, however, as you installed the kernel by apt-get install, the kernel becomes manually installed, and will not later be removed by 'sudo apt autoremove', unless you change it to be marked as automatically installed.

I question the use of 'sudo apt autoremove' in this case. If its action was the desired one, setting
Unattended-Upgrade::Remove-Unused-Dependencies "false";
would be the setting to use, instead of the current default
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Balint Reczey tries to explain in #11 why that setting is not desired.

On Thu, Sep 21, 2017 at 02:03:41PM -0000, Jarno Suni wrote:
> Brian, however, as you installed the kernel by apt-get install, the
> kernel becomes manually installed, and will not later be removed by
> 'sudo apt autoremove', unless you change it to be marked as
> automatically installed.

No, this is not the behavior I observed. After installing a new kernel
version via update-manager or 'sudo apt-get install' on Ubuntu 16.04, 'sudo
apt autoremove' does the right thing and wants to remove my third newest
kernel.

--
Brian Murray

It will remove the third newest kernel, but later, when the one you installed by "apt install" will become third newest, it will not be removed automatically. That is by design.

Jarno Suni (jarnos) wrote :

Anyway, in my view
Unattended-Upgrade::Remove-Unused-Dependencies "true";
would be the default setting to have now, and advanced users should mark the dependencies of their unpackaged software as "manual".

Brian Murray (brian-murray) wrote :

The get_auto_removable function in unattended-ugprades just checks to see if the package has the "is_auto_removable" flag set. So setting "Unattended-Upgrade::Remove-Unused-Dependencies" to "true" will produce the same outcome as using "sudo apt autoremove" e.g.:

bdmurray@clean-xenial-amd64:~$ sudo apt autoremove
[sudo] password for bdmurray:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libllvm3.8 libmircommon5 libsnapd-glib1 linux-headers-4.4.0-31 linux-headers-4.4.0-31-generic linux-image-4.4.0-31-generic
  linux-image-extra-4.4.0-31-generic snap-confine snapd-login-service ubuntu-core-launcher

bdmurray@clean-xenial-amd64:~$ sudo unattended-upgrade --dry-run
/usr/bin/dpkg --status-fd 9 --force-depends --remove libllvm3.8:amd64 libmircommon5:amd64 snapd-login-service:amd64 libsnapd-glib1:amd64 linux-headers-4.4.0-31-generic:amd64 linux-headers-4.4.0-31:all linux-image-extra-4.4.0-31-generic:amd64 linux-image-4.4.0-31-generic:amd64 snap-confine:amd64 ubuntu-core-launcher:amd64
/usr/bin/dpkg --status-fd 11 --configure --pending

So I think the question here is whether or not we think it is safe to run autoremovals without user interaction.

Brian Murray (brian-murray) wrote :

@Balint re "@Jarno: IMO Unattended-Upgrade::Remove-Unused-Dependencies is already a risky option and I don't recommend enabling it because it may remove packages which are not used according the to package-dependency chain but which users rely on using software that is not packaged."

Could you give me an example of how people would install dependencies for software that is not package that would also set the "is_auto_removable" flag in the apt cache to True? I can't think of a situation like this.

On Wed, Sep 27, 2017 at 1:49 PM, Brian Murray <email address hidden> wrote:
> @Balint re "@Jarno: IMO Unattended-Upgrade::Remove-Unused-Dependencies
> is already a risky option and I don't recommend enabling it because it
> may remove packages which are not used according the to package-
> dependency chain but which users rely on using software that is not
> packaged."
>
> Could you give me an example of how people would install dependencies
> for software that is not package that would also set the
> "is_auto_removable" flag in the apt cache to True? I can't think of a
> situation like this.

I had the opposite situation in my mind. People would remove packages
which are not auto removable, like ubuntu-desktop (for example because
something ubuntu-desktop depends on is acting badly) and then many
package in ubuntu-desktop's dependency change become auto removable:

$ sudo apt-get remove ubuntu-desktop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  a11y-profile-manager-indicator activity-log-manager adium-theme-ubuntu
  aisleriot app-install-data-partner apturl apturl-common baobab bluez-cups
...
  xorg
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  ubuntu-desktop
0 upgraded, 0 newly installed, 1 to remove and 7 not upgraded.
After this operation, 46,1 kB disk space will be freed.
Do you want to continue? [Y/n]

It is a simpler problematic situation than the one I originally mentioned, u-u
would remove software which is probably in use, no additional software is
broken.

Hmm, I thought we would not do that anymore, and packages get marked as manual when removing a meta package, but I might be missing something.

On Wed, Sep 27, 2017 at 2:42 PM, Julian Andres Klode
<email address hidden> wrote:
> Hmm, I thought we would not do that anymore, and packages get marked as
> manual when removing a meta package, but I might be missing something.

I ran this on zesty, but if meta packages are handled differently then
I could imagine a similar
case when meta packages are not involved.

Brian Murray (brian-murray) wrote :

On Wed, Sep 27, 2017 at 08:36:13PM -0000, Balint Reczey wrote:
> On Wed, Sep 27, 2017 at 2:42 PM, Julian Andres Klode
> <email address hidden> wrote:
> > Hmm, I thought we would not do that anymore, and packages get marked as
> > manual when removing a meta package, but I might be missing something.
>
> I ran this on zesty, but if meta packages are handled differently then
> I could imagine a similar
> case when meta packages are not involved.

I ran the same metapackage removal test on Xenial, so if Julian is right
and this is a regression its a rather old one. Do you have any more
details about your thoughts Julian?

--
Brian Murray

There were like 3 versions of metapackage auto removal, I don't remember exactly.

I think in the beginning, we just autoremovef everything.

Then we did not mark packages as automatically installed if installed by a meta package.

Then we moved to moving the auto bit on uninstall, but maybe only if the meta is removed due to a conflict, and not when manually.

I only followed this on the sidelines.

tags: added: id-597a833ca49ff66291d34705
Brian Murray (brian-murray) wrote :

Well this does work in Trusty, Ubuntu 14.04, perhaps this is a regression in support "Never-MarkAuto-Sections"?

One thing to note is that using 'apt-mark' I saw that vino was marked as manual on Ubuntu 14.04, but automatic on Ubuntu 16.04.

Iain Lane (laney) on 2017-10-04
Changed in gnome-software (Ubuntu Artful):
status: New → Invalid
Brian Murray (brian-murray) wrote :

I reported bug 1721364 regarding the Never-MarkAuto-Sections issue.

Balint Reczey (rbalint) on 2017-10-25
summary: - By default settings unattended-upgrade is unable to automatically remove
+ By default settings unattended-upgrade does not automatically remove
packages that become unused in conjunction with updating by other
- software.
+ software
Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
Changed in unattended-upgrades (Ubuntu Artful):
status: New → Confirmed
Changed in unattended-upgrades (Ubuntu):
status: Confirmed → Opinion
Changed in unattended-upgrades (Ubuntu Artful):
status: Confirmed → Opinion
Balint Reczey (rbalint) wrote :

Changed to title to reflect that the issue is not about something not implemented, but something not set as default.

Balint Reczey (rbalint) wrote :

s/to title/the title/

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers