linux-cloud-tools version specific packages are being removed by unattended-upgrade's Remove-Unused-Dependencies

Bug #1698159 reported by Ian Gordon on 2017-06-15
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Undecided
Unassigned
Trusty
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
unattended-upgrades (Ubuntu)
Undecided
Unassigned
Trusty
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

[Impact]
When running kernel version 4.4.0-78 (for example) and the unattended-upgrade packages installs a new kernel 4.4.0-79 (for example) the linux-cloud-tools-4.4.0-78 and linux-cloud-tools-4.4.0-78-generic packages are removed by the Remove-Unused-Dependencies rule, because unlike "normal" kernel packages they are not excluded by /etc/apt/apt.conf.d/01autoremove-kernels.

The linux-cloud-tools package has the hyper-v integration services daemons in it, so it's deletion leaves some of the integration not working.

Could /etc/apt/apt.conf.d/01autoremove-kernels be updated so other hyper-v users aren't caught out by this?

Thanks,

Ian.

[Test case (apt)]

Make sure that linux-cloud-tools for protected kernels are listed in APT::NeverAutoRemove, for example, on cosmic:

   "^linux-cloud-tools-4\.17\.0-9-generic$";
   "^linux-cloud-tools-4\.18\.0-7-generic$";

Those are in 01autoremove-kernels

[Test case (unattended-upgrades)]

 * Check the running kernel and install relevant cloud-tools packages:
# uname -a
Linux x-uu-lp-1737637 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@x-uu-lp-1737637:~# apt-cache search 4.15.0-39-generic
linux-cloud-tools-4.15.0-39-generic - Linux kernel version specific cloud tools for version 4.15.0-39
linux-headers-4.15.0-39-generic - Linux kernel headers for version 4.15.0 on 64 bit x86 SMP
linux-image-4.15.0-39-generic - Signed kernel image generic
linux-image-unsigned-4.15.0-39-generic - Linux kernel image for version 4.15.0 on 64 bit x86 SMP
linux-modules-4.15.0-39-generic - Linux kernel extra modules for version 4.15.0 on 64 bit x86 SMP
linux-modules-extra-4.15.0-39-generic - Linux kernel extra modules for version 4.15.0 on 64 bit x86 SMP
# apt install linux-cloud-tools-4.15.0-39-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  linux-cloud-tools-common linux-hwe-cloud-tools-4.15.0-39
The following NEW packages will be installed:
  linux-cloud-tools-4.15.0-39-generic linux-cloud-tools-common linux-hwe-cloud-tools-4.15.0-39
0 upgraded, 3 newly installed, 0 to remove and 17 not upgraded.
Need to get 119 kB of archives.
After this operation, 854 kB of additional disk space will be used.
Do you want to continue? [Y/n]
...

 * Mark them autoremovable and configure u-u to autoremove unused packages:

# apt-mark auto linux-cloud-tools-4.15.0-39-generic
linux-cloud-tools-4.15.0-39-generic set to automatically installed.
### set Unattended-Upgrade::Remove-Unused-Dependencies "true";
# vi /etc/apt/apt.conf.d/50unattended-upgrades

 * The not fixed version of u-u removes the packages:

# unattended-upgrade --dry-run --verbose
...
All upgrades installed
Packages that are auto removed: 'libfreetype6 linux-cloud-tools-4.15.0-39-generic linux-cloud-tools-common linux-hwe-cloud-tools-4.15.0-39'
Packages were successfully auto-removed
#

 * The fixed version does not:
 # unattended-upgrade --dry-run --verbose
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=xenial, o=Ubuntu,a=xenial-security, o=UbuntuESM,a=xenial
Removing unused kernel packages: linux-cloud-tools-common
Keeping auto-removable linux-cloud-tools-common package(s) because it would also remove the following packages which should be kept in this step: linux-cloud-tools-4.15.0-39-generic linux-hwe-cloud-tools-4.15.0-39
...

[Regression Potential]

 * some linux-cloud-tools will linger around, so it takes more space.

[Other info]

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: unattended-upgrades 0.90ubuntu0.6
ProcVersionSignature: Ubuntu 4.4.0-79.100-generic 4.4.67
Uname: Linux 4.4.0-79-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.6
Architecture: amd64
Date: Thu Jun 15 15:27:35 2017
InstallationDate: Installed on 2016-04-25 (415 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
PackageArchitecture: all
SourcePackage: unattended-upgrades
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.apt.apt.conf.d.10periodic: [modified]
modified.conffile..etc.update-motd.d.90-updates-available: [deleted]
modified.conffile..etc.update-motd.d.98-fsck-at-reboot: [deleted]
modified.conffile..etc.update-motd.d.98-reboot-required: [deleted]
mtime.conffile..etc.apt.apt.conf.d.10periodic: 2016-04-25T16:28:36.640000

Ian Gordon (ian-gordon) wrote :
Balint Reczey (rbalint) wrote :

Yes, indeed, the packages should be kept but 01autoremove-kernels is generated by /etc/kernel/postinst.d/apt-auto-removal which is shipped by apt.
Reassigning to apt.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
status: Confirmed → Triaged
affects: unattended-upgrades (Ubuntu) → apt (Ubuntu)
Balint Reczey (rbalint) on 2018-02-14
Changed in unattended-upgrades (Ubuntu):
status: New → Triaged
tags: added: rls-bb-incoming
Balint Reczey (rbalint) wrote :

I'm testing the fix for u-u right now that will rely on apt's one. The u-u fix will land soon in the git repository.

Changed in unattended-upgrades (Ubuntu):
status: Triaged → In Progress
Steve Langasek (vorlon) on 2018-02-22
tags: removed: rls-bb-incoming
tags: added: id-5a8ef52aa60dcd08c82dd6d1
Changed in apt (Ubuntu Bionic):
status: Triaged → In Progress
Julian Andres Klode (juliank) wrote :
Changed in apt (Ubuntu Bionic):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 1.0ubuntu1

---------------
unattended-upgrades (1.0ubuntu1) bionic; urgency=medium

  * Merge from Debian unstable
    - Remaining changes:
      - unattended-upgrades: Do not automatically upgrade the development
        release of Ubuntu unless Unattended-Upgrade::DevRelease is true.
    - Dropped changes, included in Debian:
      - Run upgrade-between-snapshots only on amd64.
        The test exercises only unattented-upgrade's Python code and uses
        dependencies from the frozen Debian snapshot archive thus running
        it on all architectures would provide little benefit.

unattended-upgrades (1.0) unstable; urgency=medium

  [ Simon Arlott ]
  * Revert sending mails on WARNINGS when in MailOnlyOnError mode"
  * Consider conffile prompts to be errors (Closes: #852465)
    Flag packages that have to be upgraded manually because of a conffile
    prompt and consider this to be an error when sending email or exiting.

  [ Simon McVittie ]
  * Add python, python3, setuptools, DistutilsExtra to Build-Depends.
    They are needed for `clean`, so Build-Depends-Indep is not enough.
  * Add .gitignore and debian/.gitignore
  * Remove bzr configuration.
    This is unnecessary now that u-u is in git.

  [ Michael Vogt ]
  * unattended-upgrades: tweak mail-on-warnings PR
  * unattended-upgrade: extract is_autoremove_valid helper

  [ Balint Reczey ]
  * Run upgrade-between-snapshots only on amd64.
    The test exercises only unattented-upgrade's Python code and uses
    dependencies from the frozen Debian snapshot archive thus running
    it on all architectures would provide little benefit.
  * Clean up processes started for getting md5 sums
  * Don't keep /var/lib/dpkg/status open multiple times
  * Adjust candidates in UnattendedUpgradesCache.open()
  * Perform autoremovals in minimal steps, too.
    Also add check to remove only the set of packages selected for autoremoval.
    Without that check unattended-upgrades when (by default) configured to
    remove newly unused packages could also remove auto removable packages
    which were unused before starting starting the upgrade step.
  * Remove unused automatically installed kernel packages
    (LP: #1357093, #1624644, #1675079, #1698159)
  * Stop including Python syntax in the report (Closes: #876796)
  * Do not auto remove packages related to the running kernel (LP: #1615381)
  * Check packages to be autoremoved against blacklists, whitelists.
    Also check if the packages are held.
  * Report package removals in the summary email (Closes: #876797)
  * Run upgrade-between-snapshots test with debugging enabled
  * Don't create new UnattendedUpgradesCache for checking for autoremovals
    .open() refreshes the state in each cache_commit(), this is enough
  * Update .pot and .po files
  * Update .travis.yml to actually build and test u-u from the repo
  * Run only a simple installation test on Travis, the system upgrade
    test was always failing

 -- Balint Reczey <email address hidden> Thu, 01 Mar 2018 17:29:33 +0700

Changed in unattended-upgrades (Ubuntu Bionic):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.6~beta1

---------------
apt (1.6~beta1) unstable; urgency=medium

  [ David Kalnischkies ]
  * allow the apt/lists/auxfiles/ directory to be missing (Closes: 887624)
  * add apt-helper drop-privs command…
  * restore gcc visibility=hidden for apt-private
  * ensure correct file permissions for auxfiles
  * allow the apt/lists/auxfiles/ directory to be missing (Closes: 887624)
  * add apt-helper drop-privs command…
  * restore gcc visibility=hidden for apt-private
  * ensure correct file permissions for auxfiles

  [ Julian Andres Klode ]
  * indexcopy: Copy uncompressed indices from cdrom again (LP: #1746807)
  * Work around test-method-mirror failure by setting umask at start
  * Check that Date of Release file is not in the future
  * apt.conf.autoremove: Add linux-cloud-tools to list (LP: #1698159)
  * indexcopy: Copy uncompressed indices from cdrom again (LP: #1746807)
  * Work around test-method-mirror failure by setting umask at start
  * Check that Date of Release file is not in the future
  * apt.conf.autoremove: Add linux-cloud-tools to list (LP: #1698159)

  [ Chris Leick ]
  * German manpage translation update
  * German manpage translation update

 -- Julian Andres Klode <email address hidden> Mon, 26 Feb 2018 13:14:13 +0100

Changed in apt (Ubuntu Bionic):
status: Fix Committed → Fix Released
Julian Andres Klode (juliank) wrote :

Cherry-picked the change into xenial's 1.2.y branch for the next SRU. Will do that in trusty too.

Changed in apt (Ubuntu Xenial):
status: New → Triaged
Changed in apt (Ubuntu Trusty):
status: New → Triaged
Łukasz Zemczak (sil2100) wrote :

We would need some SRU information here. Since this is part of the xenial apt SRU, we'd need to have reliable testing steps for verification of the update. Thank you!

description: updated
Łukasz Zemczak (sil2100) wrote :

Thanks! Would it be maybe possible to actually check if the cloud-tools packages aren't auto-removed by unattended-upgrades as well besides just looking at the APT::NeverAutoRemove rules?

Changed in apt (Ubuntu Xenial):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-xenial

Hello Ian, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Ian Gordon (ian-gordon) wrote :

Hi,

I installed apt 1.2.28 from proposed on a xenial server with the linux-cloud-tools packages installed and it added linux-cloud-tools to the VersionedKernelPackages section of /etc/apt/apt.conf.d/01autoremove.

It did not add linux-cloud-tools lines to the 01autoremove-kernels file. This did not occur until I upgraded the kernel to 4.4.0-137. After doing this the 01autoremove-kernels contains:

  "^linux-cloud-tools-4\.4\.0-134-generic$";
  "^linux-cloud-tools-4\.4\.0-137-generic$";

I confirmed that linux-cloud-tools package was also not auto removed.

So the new version of apt fixes the problem.

Regards,

Ian.

tags: added: verification-done-xenial
removed: verification-needed-xenial
Brian Murray (brian-murray) wrote :

Hello Ian, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.29 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed-xenial
removed: verification-done-xenial
Julian Andres Klode (juliank) wrote :

Marking verification-done-xenial again, this got reset by 1.2.29.

FWIW; I also retested: after upgrading apt to 1.2.29 in the xenial container, and rerunning the kernel hook that creates the file, I also see

   "^linux-cloud-tools-4\.18\.0-8-generic$";

in the list.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.2.29

---------------
apt (1.2.29) xenial; urgency=medium

  * Set DPKG_FRONTEND_LOCKED when running {pre,post}-invoke scripts.
    Some post-invoke scripts install packages, which fails because
    the environment variable is not set. This sets the variable for
    all three kinds of scripts {pre,post-}invoke and pre-install-pkgs,
    but we will only allow post-invoke at a later time.
    (LP: #1796808)

apt (1.2.28) xenial; urgency=medium

  [ Julian Andres Klode ]
  * apt.conf.autoremove: Add linux-cloud-tools to list (LP: #1698159)
  * Add support for dpkg frontend lock (Closes: #869546) (LP: #1781169)
  * Set DPKG_FRONTEND_LOCKED as needed when doing selection changes
  * http: Stop pipeline after close only if it was not filled before
    (LP: #1794957)
  * pkgCacheFile: Only unlock in destructor if locked before (LP: #1794053)
  * Update libapt-pkg5.0 symbols for frontend locking

  [ David Kalnischkies ]
  * Support records larger than 32kb in 'apt show' (Closes: #905527)
    (LP: #1787120)

 -- Julian Andres Klode <email address hidden> Tue, 09 Oct 2018 12:35:03 +0200

Changed in apt (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Balint Reczey (rbalint) on 2018-11-30
description: updated

Hello Ian, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
removed: verification-done verification-done-xenial
Łukasz Zemczak (sil2100) wrote :

Hello Ian, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers