/usr/bin/unattended-upgrade:apt.cache.LockFailedException:/usr/bin/unattended-upgrade@1468:main:do_auto_remove:cache_commit:commit:_fetch_archives

Bug #1602536 reported by errors.ubuntu.com bug bridge on 2016-07-13
80
This bug affects 14 people
Affects Status Importance Assigned to Milestone
python-apt (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
unattended-upgrades (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

[Impact]

 * Unattended-upgrades fails to autoremove packages after installing updates or fails to install updates due to a parallel process acquiring apt or dpkg lock while u-u is running.

[Test Case]

 * Set up a system with packages (> 30) to be upgraded. In case of Bionic -security has a low number of updates, thus set up the system to install -updates to have more packages for u-u to upgrade:

 echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51-updates

 * Set up two shells to run commands in parallel
 * In shell "A" run sudo apt update && sudo unattended-upgrade --dry-run --verbose --debug
 * After u-u started run the following command in shell "B":
 while sleep 1; do python3 -c 'import apt; import apt_pkg; print(apt_pkg.pkgsystem_lock())' ; done
 * Observe the following exception repeated while running u-u and True being printed after u-u is finished:
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
True
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
True
True
True
True
...

[Regression Potential WIP]

 * Unattended-upgrade may crash

[Other Info]

It looks like python-apt also releases the lock sometimes unexpectedly thus the both packages need to be fixed to avoid loosing the lock.

[Original Bug Text]

The Ubuntu Error Tracker has been receiving reports about a problem regarding unattended-upgrades. This problem was most recently seen with version 0.90, the problem page at https://errors.ubuntu.com/problem/19f99745d7dce5aea118eb31bfffcbdcdf238c4f contains more details.

Daniel van Vugt (vanvugt) wrote :

Apparently this is the hottest crash in Ubuntu for the past year...
https://errors.ubuntu.com/?period=year

Adolfo Jayme (fitojb) on 2016-09-20
Changed in unattended-upgrades (Ubuntu):
importance: Undecided → High
Brian Murray (brian-murray) wrote :

Here is the Traceback:

Traceback (most recent call last):
  File "/usr/bin/unattended-upgrade", line 1468, in <module>
    main(options)
  File "/usr/bin/unattended-upgrade", line 1397, in main
    options.verbose or options.debug)
  File "/usr/bin/unattended-upgrade", line 1106, in do_auto_remove
    res, error = cache_commit(cache, logfile_dpkg, verbose)
  File "/usr/bin/unattended-upgrade", line 408, in cache_commit
    res = cache.commit(install_progress=iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 512, in commit
    res = self._fetch_archives(fetcher, pm)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 338, in _fetch_archives
    raise LockFailedException("Failed to lock %s" % lockfile)
apt.cache.LockFailedException: Failed to lock /var/cache/apt/archives/lock

Brian Murray (brian-murray) wrote :

I'd guess there is another package manager running when we the unattended-upgrade job gets called and unlocks the package system. Although its odd the crashes are all about do_auto_remove.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
Sebastian Unger (sebunger44) wrote :

I believe this happens because unattended-upgrades gets and releases the lock a couple times (such as between installing upgrades and then removing now no longer required automatically installed packages) and dies if it cannot re-aquire the lock (rather than gracefully handling it).

The high frequency is easily explained since apt-daily runs both unattended-upgrades and aptd (via dbus triggers both from Update-Post-Invoke-Success (or whatever it was called) and directly from the apt-daily script) and this sets up a race with a fairly high chance of hitting unattended-upgrades somewhere where it hurts.

Even when it does not lead to a crash it often causes upgrades not to be installed. This can be seen by setting APT::Periodic::Verbose to 3 and monitoring the logs. Often one or another of the steps of apt-daily will fail because it could not acquire the lock.

Hope this helps.

Balint Reczey (rbalint) on 2018-06-07
description: updated
Balint Reczey (rbalint) wrote :

U-u is fixed in version 1.2.

description: updated
Changed in unattended-upgrades (Ubuntu):
status: Confirmed → Fix Released

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Changed in python-apt (Ubuntu):
status: New → Fix Committed
Balint Reczey (rbalint) wrote :
Download full text (3.4 KiB)

Verified with 1.1ubuntu1.18.04.1:

Log from shell "A":
rbalint@xenial-test:~$ lxc exec bb-uu-fixed -- bash
root@bb-uu-fixed:~# unattended-upgrade --dry-run --verbose
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: libgcrypt20 libssl1.0.0 libssl1.1 openssl
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libgcrypt20_1.8.1-4ubuntu1.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --no-triggers --configure libgcrypt20:amd64
/usr/bin/dpkg --status-fd 9 --configure --pending
Preconfiguring packages ...
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libssl1.0.0_1.0.2n-1ubuntu5.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --configure --pending
Preconfiguring packages ...
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/libssl1.1_1.1.0g-2ubuntu4.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --configure --pending
/usr/bin/dpkg --status-fd 9 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/openssl_1.1.0g-2ubuntu4.1_amd64.deb
/usr/bin/dpkg --status-fd 9 --configure --pending
All upgrades installed

Attaching log for shell "B", since it is quite long.
In the attached log it can be observed that the lock is still lost by u-u for small windows (due to python-apt losing it), but u-u then reacquires it, which is what the fix for u-u should do.

Unfixed unattended-upgrades 1.1ubuntu1, shell "B":

root@bb-uu:~# while sleep 0.3; do python3 -c 'import apt; import apt_pkg; print(apt_pkg.pkgsystem_lock())' ; done
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Traceback (most recent call last):
  File "<string>", line 1, in <module>
apt_pkg.Error: E:Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable), E:Unable to ...

Read more...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Brian Murray (brian-murray) wrote :

The bucket for the 18.04 version of this crash, https://errors.ubuntu.com/problem/33db55f3a084a7957ca1ec76f7ac2870614f17fa, only contains a few instances of the new package version and that's likely due to apport gathering the package version number after the crash occurs.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.1

---------------
unattended-upgrades (1.1ubuntu1.18.04.1) bionic; urgency=medium

  [ Michael Vogt ]
  * unattended-upgrades: fix Unlocked context manager. (LP: #1602536)
    The Unlocked context manager did correctly unlock but did not
    reacquire the lock which means that in minimal-upgrade step
    mode it is possible to run apt code without a lock. If something
    else (like landscape, apt, synaptic, packagekit) locks the cache
    in the meantime this will work and u-u will get dpkg errors
    because dpkg will not be able to perform its operations. It is
    less of an issue in non-minimal mode, but even then the auto-remove
    step may fail in this way.

  [ Balint Reczey ]
  * Fix adjusting candidates (LP: #1775292)
  * Relock apt lock before reopening the cache (LP: #1602536)
  * Fix crashing while adjusting candidates and save candidates to adjust only
    in first sweep run, not emptying the set later
    (Closes: #901258) (LP: #1775307)

 -- Balint Reczey <email address hidden> Wed, 06 Jun 2018 16:30:55 -0700

Changed in unattended-upgrades (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
removed: verification-done
Łukasz Zemczak (sil2100) wrote :

Hello errors.ubuntu.com, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers