cache.commit() doesn't release the archives lock

[Julian Andres Klode]

[Impact]
cache.commit() does not release all the locks it acquires as part of committing. This is a regression of locking fixes in bug 1795407.

[Test case]
This script should work:

#!/usr/bin/env python
import apt
import subprocess

cache = apt.Cache()
pkg = cache["hello"]
pkg.mark_install()
cache.commit()
subprocess.check_call(["apt", "remove", "--yes", "hello"])

[Regression potential]
Other new locking bugs could pop up

[Julian Andres Klode (juliank)]

This was fixed in disco, in 1.8.4. It also affects cosmic and bionic; and the proposed SRU for bug 1795407 in xenial.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.6.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted python-apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.1.0~beta1ubuntu0.16.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

I installed the upgrades, and ran the script in the bug description and see if it got fixed.

xenial python3-apt (1.1.0~beta1ubuntu0.16.04.4) verified - note that the one in -updates is fine too, it only broke for the previous upload in proposed.
bionic python3-apt (1.6.4) verified
cosmic python3-apt (1.7.1) verified

Full testing script for inside a lxd container:

while ! grep cloud-init /etc/apt/sources.list; do
 sleep 1
done
while ! ping -c1 google.com; do
 sleep 1
done
echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -cs)-proposed main" >> /etc/apt/sources.list
apt update
python3 - << EOF
#!/usr/bin/env python
import apt
import subprocess

cache = apt.Cache()
pkg = cache["hello"]
pkg.mark_install()
cache.commit()
subprocess.check_call(["apt", "remove", "--yes", "hello"])
EOF
apt install python3-apt
python3 - << EOF
#!/usr/bin/env python
import apt
import subprocess

cache = apt.Cache()
pkg = cache["hello"]
pkg.mark_install()
cache.commit()
subprocess.check_call(["apt", "remove", "--yes", "hello"])
EOF

Run like this:

( i=cosmic; lxc launch -e ubuntu:$i test-$i && lxc exec test-$i -- bash -x - < ~/a.sh ; lxc stop test-$i )

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.7.1

---------------
python-apt (1.7.1) cosmic; urgency=medium

  * apt.Cache: Fix (un)locking of archives (Closes: #922416) (LP: #1826870)
  * apt.Cache: Use explicit, more safe locking in update()
  * Update mirror lists

 -- Julian Andres Klode <email address hidden> Mon, 29 Apr 2019 13:44:16 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for python-apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Both the bionic and xenial uploads seems to have a few ADT regressions - could you take at those and see if they're not regressions? Thanks!

[Julian Andres Klode (juliank)]

xenial:
- ubuntu-make not a regression, same bug as before
- apport retried
- snapcraft not a regression, same issues as before

[Julian Andres Klode (juliank)]

xenial: None of the failures are regressions; the apport one we can also see just with apport as trigger.

[Julian Andres Klode (juliank)]

bionic: Only apt-clone and snapcraft left after retries. Retried them both with themselves as triggers, and they're still failing, so no regressions either.

[Brian Murray (brian-murray)]

I don't see apt-clone having been tried with itself as a trigger for amd64.

http://autopkgtest.ubuntu.com/packages/a/apt-clone/bionic/amd64

[Julian Andres Klode (juliank)]

Well, no, I'm not going to waste resources retrying all the archs, I only retried ppc64el.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.6.4

---------------
python-apt (1.6.4) bionic; urgency=medium

  * apt.Cache: Fix (un)locking of archives (Closes: #922416) (LP: #1826870)
  * apt.Cache: Use explicit, more safe locking in update()
  * Update mirror lists

 -- Julian Andres Klode <email address hidden> Mon, 29 Apr 2019 13:49:15 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.1.0~beta1ubuntu0.16.04.4

---------------
python-apt (1.1.0~beta1ubuntu0.16.04.4) xenial; urgency=medium

  * apt.Cache: Fix (un)locking of archives (Closes: #922416) (LP: #1826870)
  * apt.Cache: Use explicit, more safe locking in update()
  * Update mirror lists

python-apt (1.1.0~beta1ubuntu0.16.04.3) xenial; urgency=medium

  * Frontend locking and related locking improvements (LP: #1795407)
    - apt.Cache: Keep / Re-establish the system lock in commit()
    - apt.Cache: Keep archive locked during commit()/ in fetch_archives()
    - apt.Cache: Reinstate locks in a finally / run dpkg inside try
    - Introduce frontend locking
    - Convert apt.Cache.commit and apt_pkg.DepCache.commit to FE lock
  * Other changes to make that work:
    - travis CI: enable PPA for new apt versions
  * As always, updated mirror lists

 -- Julian Andres Klode <email address hidden> Mon, 29 Apr 2019 14:11:32 +0200