/usr/bin/unattended-upgrade:apt.cache.LockFailedException:/usr/bin/unattended-upgrade@1468:main:do_auto_remove:cache_commit:commit:_fetch_archives

Bug #1602536 reported by errors.ubuntu.com bug bridge
72
This bug affects 14 people
Affects Status Importance Assigned to Milestone
unattended-upgrades (Ubuntu)
Fix Released
High
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * Unattended-upgrades fails to autoremove packages after installing updates or fails to install updates due to a parallel process acquiring apt or dpkg lock while u-u is running.

[Test Case]

 * Set up a system with packages (> 30) to be upgraded. In case of Bionic -security has a low number of updates, thus set up the system to install -updates to have more packages for u-u to upgrade:

 echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51-updates

 * Set up two shells to run commands in parallel
 * In shell "A" run sudo apt update && sudo unattended-upgrade --dry-run --verbose --debug
 * After u-u started run the following command in shell "B":
 while sleep 1; do python3 -c 'import apt; import apt_pkg; print(apt_pkg.pkgsystem_lock())' ; done
 * Observe the following exception repeated while running u-u and True being printed after u-u is finished:
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
True
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
True
True
True
True
...

[Regression Potential WIP]

 * Unattended-upgrade may crash

[Other Info]

It looks like python-apt also releases the lock sometimes unexpectedly thus the both packages need to be fixed to avoid loosing the lock.

[Original Bug Text]

The Ubuntu Error Tracker has been receiving reports about a problem regarding unattended-upgrades. This problem was most recently seen with version 0.90, the problem page at https://errors.ubuntu.com/problem/19f99745d7dce5aea118eb31bfffcbdcdf238c4f contains more details.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Apparently this is the hottest crash in Ubuntu for the past year...
https://errors.ubuntu.com/?period=year

Changed in unattended-upgrades (Ubuntu):
importance: Undecided → High
Revision history for this message
Brian Murray (brian-murray) wrote :

Here is the Traceback:

Traceback (most recent call last):
  File "/usr/bin/unattended-upgrade", line 1468, in <module>
    main(options)
  File "/usr/bin/unattended-upgrade", line 1397, in main
    options.verbose or options.debug)
  File "/usr/bin/unattended-upgrade", line 1106, in do_auto_remove
    res, error = cache_commit(cache, logfile_dpkg, verbose)
  File "/usr/bin/unattended-upgrade", line 408, in cache_commit
    res = cache.commit(install_progress=iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 512, in commit
    res = self._fetch_archives(fetcher, pm)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 338, in _fetch_archives
    raise LockFailedException("Failed to lock %s" % lockfile)
apt.cache.LockFailedException: Failed to lock /var/cache/apt/archives/lock

Revision history for this message
Brian Murray (brian-murray) wrote :

I'd guess there is another package manager running when we the unattended-upgrade job gets called and unlocks the package system. Although its odd the crashes are all about do_auto_remove.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
Revision history for this message
Sebastian Unger (sebunger44) wrote :

I believe this happens because unattended-upgrades gets and releases the lock a couple times (such as between installing upgrades and then removing now no longer required automatically installed packages) and dies if it cannot re-aquire the lock (rather than gracefully handling it).

The high frequency is easily explained since apt-daily runs both unattended-upgrades and aptd (via dbus triggers both from Update-Post-Invoke-Success (or whatever it was called) and directly from the apt-daily script) and this sets up a race with a fairly high chance of hitting unattended-upgrades somewhere where it hurts.

Even when it does not lead to a crash it often causes upgrades not to be installed. This can be seen by setting APT::Periodic::Verbose to 3 and monitoring the logs. Often one or another of the steps of apt-daily will fail because it could not acquire the lock.

Hope this helps.

Balint Reczey (rbalint)
description: updated
Revision history for this message
Balint Reczey (rbalint) wrote :

U-u is fixed in version 1.2.

description: updated
Changed in unattended-upgrades (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Changed in python-apt (Ubuntu):
status: New → Fix Committed
Revision history for this message
Balint Reczey (rbalint) wrote :
Download full text (3.4 KiB)

Verified with 1.1ubuntu1.18.04.1:

Log from shell "A":
rbalint@xenial-test:~$ lxc exec bb-uu-fixed -- bash
root@bb-uu-fixed:~# unattended-upgrade --dry-run --verbose
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: libgcrypt20 libssl1.0.0 libssl1.1 openssl
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libgcrypt20_1.8.1-4ubuntu1.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --no-triggers --configure libgcrypt20:amd64
/usr/bin/dpkg --status-fd 9 --configure --pending
Preconfiguring packages ...
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libssl1.0.0_1.0.2n-1ubuntu5.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --configure --pending
Preconfiguring packages ...
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libssl1.1_1.1.0g-2ubuntu4.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --configure --pending
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/openssl_1.1.0g-2ubuntu4.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --configure --pending
All upgrades installed

Attaching log for shell "B", since it is quite long.
In the attached log it can be observed that the lock is still lost by u-u for small windows (due to python-apt losing it), but u-u then reacquires it, which is what the fix for u-u should do.

Unfixed unattended-upgrades 1.1ubuntu1, shell "B":

root@bb-uu:~# while sleep 0.3; do python3 -c 'import apt; import apt_pkg; print(apt_pkg.pkgsystem_lock())' ; done
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to ...

Read more...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

The bucket for the 18.04 version of this crash, https://errors.ubuntu.com/problem/33db55f3a084a7957ca1ec76f7ac2870614f17fa, only contains a few instances of the new package version and that's likely due to apport gathering the package version number after the crash occurs.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.1

---------------
unattended-upgrades (1.1ubuntu1.18.04.1) bionic; urgency=medium

  [ Michael Vogt ]
  * unattended-upgrades: fix Unlocked context manager. (LP: #1602536)
    The Unlocked context manager did correctly unlock but did not
    reacquire the lock which means that in minimal-upgrade step
    mode it is possible to run apt code without a lock. If something
    else (like landscape, apt, synaptic, packagekit) locks the cache
    in the meantime this will work and u-u will get dpkg errors
    because dpkg will not be able to perform its operations. It is
    less of an issue in non-minimal mode, but even then the auto-remove
    step may fail in this way.

  [ Balint Reczey ]
  * Fix adjusting candidates (LP: #1775292)
  * Relock apt lock before reopening the cache (LP: #1602536)
  * Fix crashing while adjusting candidates and save candidates to adjust only
    in first sweep run, not emptying the set later
    (Closes: #901258) (LP: #1775307)

 -- Balint Reczey <email address hidden> Wed, 06 Jun 2018 16:30:55 -0700

Changed in unattended-upgrades (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
removed: verification-done
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (33.9 KiB)

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2

---------------
unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium

  * Don't check blacklist too early and report updates from not allowed origins
    as kept back. (LP: #1781176)
  * test/test_blacklisted_wrong_origin.py: Fix and enable test
  * Filter out progress indicator from dpkg log (LP: #1599646)
  * Clear cache when autoremoval fails (LP: #1779157)
  * Find autoremovable kernel packages using the patterns in APT's way
    (LP: #1815494)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium

  * Start service after systemd-logind.service to be able to take inhibition
    lock (LP: #1806487)
  * Handle gracefully when logind is down (LP: #1806487)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium

  * Backport to Xenial (LP: #1702793)
  * Revert to build-depending on debhelper (>= 9~) and dh-systemd
  * Revert configuration example changes to avoid triggering a debconf question
  * debian/postinst: Update recovery to be triggered on Xenial's package versions

unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium

  * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
    Performing upgrades in service's ExecStop did not work when the upgrades
    involved restarting services because systemd blocked other stop/start
    actions making maintainer scripts time out and be killed leaving a broken
    system behind.
    Running unattended-upgrades.service before shutdown.target as a oneshot
    service made it run after unmounting filesystems and scheduling services
    properly on shutdown is a complex problem and adding more services to the
    mix make it even more fragile.
    The solution of monitoring PrepareForShutdown() signal from DBus
    allows Unattended Upgrade to run _before_ the jobs related to shutdown are
    queued thus package upgrades can safely restart services without
    risking causing deadlocks or breaking part of the shutdown actions.
    Also ask running unattended-upgrades to stop when shutdown starts even in
    InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
    UnattendedUpgradesShutdown class. (LP: #1778219)
  * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219)
    This allows more time for unattended-upgrades to shut down gracefully
    or even install a few packages in InstallOnShutdown mode, but is still a
    big step back from the 30 minutes allowed for InstallOnShutdown previously.
    Users enabling InstallOnShutdown node are advised to increase
    InhibitDelayMaxSec even further possibly to 30 minutes.
    - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
      changes
  * Ignore "W503 line break before binary operator"
    because it will become the best practice and breaks the build
  * Stop using ActionGroups, they interfere with apt.Cache.clear()
    causing all autoremovable packages to be handled as newly autoremovable
    ones and be removed by default. Dropping ActionGroup usage does not slow
    down the most frequent case of not having anything to upgrade a...

Changed in unattended-upgrades (Ubuntu Xenial):
status: Fix Committed → Fix Released
Mathew Hodson (mhodson)
no longer affects: python-apt (Ubuntu Bionic)
no longer affects: python-apt (Ubuntu Xenial)
no longer affects: python-apt (Ubuntu)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.