checking trust of archives eats a lot of cpu

Bug #1396787 reported by Ernst Kloppenburg on 2014-11-26
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
unattended-upgrades (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

[Impact]

 * Unattended-upgrades consumes tens of seconds or even minutes of CPU time to verify the origin of the packages

 * Using excessive amount of CPU is unpleasant for desktop/laptop users and also wastes computation time on servers/cloud instances.

 * Unattended-upgrades' algorithm for checking and adjusting package origins is redesigned to visit and adjust less packages.

[Test Case]

 * The added upgrade-all-security autopkgtest measure the time u-u needs for upgrading security updates on the tested release starting with no security updates applied to the point where all security updates are applied but all packages are left upgradable from <release>-updates. The test also measures the time needed for --dry-run to find no updates to be installed unattended.
* Please run autopkgtests and look for the to time results:
...
All upgrades installed
44.41user 3.06system 0:48.35elapsed 98%CPU (0avgtext+0avgdata 164872maxresident)k
208inputs+192376outputs (0major+642657minor)pagefaults 0swaps
...
No packages found that can be upgraded unattended and no pending auto-removals
2.83user 0.11system 0:02.98elapsed 98%CPU (0avgtext+0avgdata 79308maxresident)k

[Regression Potential]

 * Due to algorithm redesign there is a risk that packages from allowed origins are not upgraded. There were unit tests for testing the selection of the right packages to upgrade already, but a new autopkgtest is also introduce to verify u-u's behavior on current real-life security-updates.

[Original bug text]

(System: Ubuntu 14.04, up to date packages)

I noticed that unattended-upgrades spends a significant amount of time in phases where it runs at 100% cpu. On a slower machine (core 2 t7200 2GHz) this goes on for minutes rather than seconds. This interferes with using the machine for other tasks.

Using the --debug option to unattended-upgrades shows that the program outputs a lot of lines like the following during these 100% cpu phases:

matching 'a'='trusty-updates' against '<Origin component:'universe' archive:'trusty-updates' origin:'Ubuntu' label:'Ubuntu' site:'de.archive.ubuntu.com' isTrusted:True>

From this output I guess the operation executed is not so complicated that it should require so much cpu power. ??

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: unattended-upgrades 0.82.1ubuntu2
ProcVersionSignature: Ubuntu 3.13.0-40.69-generic 3.13.11.10
Uname: Linux 3.13.0-40-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
Date: Wed Nov 26 21:53:57 2014
InstallationDate: Installed on 2014-08-28 (90 days ago)
InstallationMedia: Kubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.1)
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SourcePackage: unattended-upgrades
UpgradeStatus: No upgrade log present (probably fresh install)

Trent Waddington (qg) wrote :

Is there a generic bug or effort for the obtrusiveness of unattended-upgrades?

umiyosh (yumino99) wrote :

I have same problem on GCP VM(n1-standard-2 and g1-small).
a result of dstat indicate that xz and python3 consume most of cpu time on the task.
I think this is fatal thing on small core machine.

Swistak (swistakers) wrote :

I've hit the same bug and did some investigation. I profiled the unattended-upgrade script with python's cProfile and attached the result.

I've run:
sudo python3 -m cProfile -s time /usr/bin/unattended-upgrade -d -v
on Ubuntu 16.04.3 booted from live cd after uncommenting the "${distro_id}:${distro_codename}-security"; line in /etc/apt/apt.conf.d/50unattended-upgrades. Otherwise no packages are selected for an upgrade and the issue doesn't reproduce.

It turns out that several functions in apt's cache are called millions of times. I tracked it down to the quadratic nature of the algorithm, which appears to come from the very initial version of the script:
https://github.com/mvo5/unattended-upgrades/blob/0ec001874ad48300199565f8591818ca75bc5f9f/unattended.py#L47

The check_changes_for_sanity() function loops over all packages in the cache. This function is called for every package marked for upgrade.

The problem appears on systems with many packages to upgrade (in my case it was 169). It makes the initial unattended-upgrades run ridiculously slow. Also, on systems configured to only perform security updates, but not other updates there's always a considerable list of packages to upgrade. It renders unattended-upgrades almost unusable on such systems.

I think this is a serious issue and the algorithm should be overhauled. Doing so many checks seems unnecessary.

Swistak (swistakers) wrote :

After digging a bit deeper I found out that that the default configuration of /etc/apt/apt.conf.d/50unattended-upgrades is exactly the same as I had in my experiment i.e.
- the ${distro_id}:${distro_codename}-security origin is ENABLED
- the ${distro_id}:${distro_codename}-updates and others origin are DISABLED

The consequences are the following. A fresh install of Ubuntu LTS e.g. 16.04 keeps installing the *-security updates, but not the *-updates ones. The number of packages with new versions in *-updates repository keeps growing unless somebody installs them manually. It is a couple of clicks, but still many people never do it. As the list of upgradable packages grows, unattended-upgrades get slower and slower to the point that it may take hours to complete. Effectively, the automatic updates may no longer work at all depending on how a given system is used. Also the user experience gets worse with CPU busy executing the unattended-upgrades script.

All in all, I consider it a serious issue that requires attention.

Balint Reczey (rbalint) wrote :

Please note that latest unattended-upgrades contain several speed improvements which are planned to be back-ported to Xenial (and Trusty).

tags: added: id-5acfd3863b9f75d6e6f2a6df
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 1.4ubuntu1

---------------
unattended-upgrades (1.4ubuntu1) cosmic; urgency=medium

  * Merge from Debian unstable
    - Remaining changes:
      - unattended-upgrades: Do not automatically upgrade the development
        release of Ubuntu unless Unattended-Upgrade::DevRelease is true.
    - Dropped changes, included in Debian:
      - Stop rewriting test apt.conf in test_untrusted.py.
      - Fix test_on_battery on Ubuntu development releases

unattended-upgrades (1.4) unstable; urgency=medium

  * Skip starting init.d script in debhelper-generated postinst part
    (LP: #1778800)
  * Use "deb-systemd-invoke start" instead of "systemctl start" in postinst.
    It is used only in a workaround applied for a Debian bug and for upgrading
    from pre-bionic versions.
  * Clear cache when autoremoval is invalid for a package set marked for
    removal (LP: #1779157)
  * Clear cache after failed commits to return from a possibly invalid state
  * Unlock for dpkg operations with apt_pkg.pkgsystem_unlock_inner() when it
    is available, also stop running when reacquiring the lock fails.
    Thanks to to Julian Andres Klode for original partial patch
  * Use fully qualified domain name in email subject.
  * Send email about all failures and crashes (Closes: #898607)
  * Add short textual summary of the results in the summary email
  * Recommend overriding configuration in a separate file.
    This can be better than changing /etc/apt/apt.conf.d/50unattended-upgrades
    because package updates don't conflict with local changes this way.
  * Adjust candidates only for packages to be possibly installed
    (Closes: #892028, #899366) (LP: #1396787)
  * Add Unattended-Upgrade::OnlyOnACPower config file example for all
    distributions
  * debian/control: Drop redundant Testsuite: autopkgtest field to keep Lintian
    happy
  * Bump Standards-Version to 4.1.4
  * Add debian/tests/upgrade-all-security to install all current security
    updates. On development releases this tests latest stable, on stable
    releases it tests the release itself.
  * Skip updates on metered connections (Closes: #855570)
  * Quote shell variables in autopkgtest
  * Stop u-u early when it should stop later anyway
  * Measure time for --dry-run and after all updates are installed in
    autopkgtests
  * Filter out packages cheaper when they are not from allowed origins
  * Collect autoremovable packages, too, when looking for upgradable ones

 -- Balint Reczey <email address hidden> Fri, 06 Jul 2018 13:33:29 +0200

Changed in unattended-upgrades (Ubuntu):
status: New → Fix Released
Balint Reczey (rbalint) on 2018-07-11
description: updated

Hello Ernst, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Steve Langasek (vorlon) wrote :

Hello Ernst, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Łukasz Zemczak (sil2100) wrote :

Hello Ernst, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Balint Reczey (rbalint) wrote :

With 1.1ubuntu1.18.04.3 u-u runs in ~3s when no package should be updated but there are upgradable packages in -updates.

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/u/unattended-upgrades/20180713_084216_9f8ee@/log.gz :
...
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
No packages found that can be upgraded unattended and no pending auto-removals
2.71user 0.12system 0:02.84elapsed 99%CPU (0avgtext+0avgdata 78924maxresident)k
0inputs+7736outputs (0major+22416minor)pagefaults 0swaps
...

The test for 1.1ubuntu1.18.04.4 was most likely run on a slower system where the same run takes ~5s.

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/u/unattended-upgrades/20180713_124827_de117@/log.gz :
...
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
No packages found that can be upgraded unattended and no pending auto-removals
5.01user 0.14system 0:05.16elapsed 99%CPU (0avgtext+0avgdata 78992maxresident)k
0inputs+7736outputs (0major+22427minor)pagefaults 0swaps
...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.4

---------------
unattended-upgrades (1.1ubuntu1.18.04.4) bionic; urgency=medium

  * Redirect stderr output in upgrade-between-snapshots, too, otherwise it
    breaks the test sometimes (LP: #1781446)

unattended-upgrades (1.1ubuntu1.18.04.3) bionic; urgency=medium

  * Redirect stderr output in upgrade-all-security, otherwise it breaks the
    test (LP: #1781446)

unattended-upgrades (1.1ubuntu1.18.04.2) bionic; urgency=medium

  [ Balint Reczey ]
  * Clear cache when autoremoval is invalid for a package set marked for
    removal and clear cache after failed commits to return from a possibly
    invalid state (LP: #1779157)
  * Don't start or gracefully stop upgrade on battery (LP: #1773033)
  * Skip updates on metered connections (Closes: #855570) (LP: #1781183)
  * Add debian/tests/upgrade-all-security to install all current security updates.
    On development releases this tests latest stable, on stable releases it tests
    the release itself.
  * Speed up unattended-upgrade (Closes: #892028, #899366) (LP: #1396787)
    - Adjust candidates only for packages to be possibly installed
    - Filter out packages cheaper when they are not from allowed origins
    - Collect autoremovable packages, too, when looking for upgradable ones
    - Measure time of running with --dry-run in autopkgtests
  * Skip starting init.d script in debhelper-generated postinst part
    (LP: #1778800)

  [ Ivan Kurnosov ]
  * Fixed is_pkgname_in_blacklist to be side-effect free. (LP: #1781176)
    Otherwise 'is_pkgname_in_blacklist' mutates the 'pkgs_kept_back' and
    'unattended-upgrades' treats the package as a blacklisted candidate

 -- Balint Reczey <email address hidden> Fri, 13 Jul 2018 10:36:23 +0200

Changed in unattended-upgrades (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Ernst, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
removed: verification-done
Łukasz Zemczak (sil2100) wrote :

Hello Ernst, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers