python-apt frontend locking

Bug #1795407 reported by Julian Andres Klode
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python-apt (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
Add support to python-apt for frontend locking. This is a bit more complicated, and also requires some other restructuring:

(1) The archives lock was only taken for a short time, when it should have been kept for the duration of an installation, as otherwise debs could disappear from archives/ while the install is running.

(2) There was no lock handling at all. Tools essentially acquire the lock, and then release it before commit().

The fix is based on the apt fix, which makes apt_pkg.SystemLock() manage both the frontend and the normal lock, and allows code to call apt_pkg.pkgsystem_unlock_inner() and apt_pkg.pkgsystem_lock_inner() around dpkg invocations to release the inner lock.

Cache.commit() takes care of locking itself now. It releases the inner lock as needed for dpkg invocations. It also now requires apt_pkg.SystemLock to be hold - if it is not, it will acquire it itself.

[Test case]
1. Mark a package in the cache for install/removal and commit() - FE lock should be taken while dpkg is running.
2. Mark a package in the cache for install/removal and commit() while holding the lock. Releasing the lock afterwards should still work correctly, and the FE lock should never be released
3. Observe that the archive lock is not released until the last dpkg run

The locking behavior can be observed via strace.

[Regression potential]
The dpkg lock changed the most: Some lock counting might go wrong and programs might fail as a result. Any problems will be isolated, though - dpkg still acquires its locks, and if a lock miscount happens, apt would not tell dpkg to skip acquiring the frontend lock, so even if we were losing the lock anywhere, dpkg would still try to acquire it and not run unlocked.

The archive lock done by commit()/fetch_archives() in apt.Cache() is now hold as long as the fetcher object exists though (as it uses the fetcher's get_lock method), which might cause unexpected behavior in apps not expecting that.

The lists/ lock is unaffected by the changes, so there should not be any regressions when it comes to updating index files.

[Other info (1)]
CI changes: There were some improvements to type hints for mypy needed to make type checks pass, and changes to the CI's Dockerfile to pull a new apt from the apt stable PPA (as a new enough apt is only in unapproved atm). Neither of those files are used anywhere outside of the package, so they should not cause any problems.

[Other info (2)]
This is part of a wider series of SRUs for frontend locking

- dpkg (bug 1796081)
- apt (bug 1781169)
- python-apt (bug 1795407)
- packagekit (bug 1795614)
- unattended-upgrades (bug 1789637)
- aptdaemon (no bug filed yet)

Further details about frontend locking can be found in https://lists.debian.org/debian-dpkg/2017/01/msg00044.html

- This SRU depends on the apt SRU above, and breaks unattended-upgrades in bionic without its SRU due to a bug in the u-u code in handling the new API.

- xenial depends on unattended-upgrades SRU

Changed in python-apt (Ubuntu):
status: New → Fix Released
Changed in python-apt (Ubuntu Xenial):
status: New → Triaged
Changed in python-apt (Ubuntu Bionic):
status: New → Triaged
description: updated
description: updated
Changed in python-apt (Ubuntu Xenial):
status: Triaged → In Progress
Changed in python-apt (Ubuntu Bionic):
status: Triaged → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted python-apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.6.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-apt (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Julian, or anyone else affected,

Accepted python-apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.1.0~beta1ubuntu0.16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-apt (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
tags: added: id-5bae2d1cfc953731f10adda4
description: updated
description: updated
description: updated
Revision history for this message
Julian Andres Klode (juliank) wrote :

bionic 1.6.3:

1. Verified that the lock is hold during dpkg run added lslocks to aptitude prerm:

$ lxc exec bbb -- strace -o $PWD/log -e openat,open,close,fcntl python3 $PWD/foo.py
(Reading database ... 28970 files and directories currently installed.)
Removing aptitude (0.8.10-6ubuntu1) ...
COMMAND PID TYPE SIZE MODE M START END PATH
atd 198 POSIX 4B WRITE 0 0 0 /run/atd.pid
python3 2751 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock-frontend
python3 2751 POSIX 0B WRITE 0 0 0 /var/cache/apt/archives/lock
cron 196 FLOCK 4B WRITE 0 0 0 /run/crond.pid
dpkg 2761 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock
Removing aptitude-common (0.8.10-6ubuntu1) ...
Processing triggers for man-db (2.8.3-2) ...

2. With system lock helt in my script:

#!/usr/bin/python3
import apt, apt_pkg

c = apt.Cache()
if c["aptitude-common"].is_installed:
 c["aptitude-common"].mark_delete()
else:
 c["aptitude-common"].mark_install()

with apt_pkg.SystemLock():
 c.commit()

$ lxc exec bbb -- strace -o $PWD/log.locked -e openat,open,close,fcntl python3 $PWD/foo.py
(Reading database ... 28970 files and directories currently installed.)
Removing aptitude (0.8.10-6ubuntu1) ...
COMMAND PID TYPE SIZE MODE M START END PATH
atd 198 POSIX 4B WRITE 0 0 0 /run/atd.pid
dpkg 3106 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock
cron 196 FLOCK 4B WRITE 0 0 0 /run/crond.pid
python3 3096 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock-frontend
python3 3096 POSIX 0B WRITE 0 0 0 /var/cache/apt/archives/lock
Removing aptitude-common (0.8.10-6ubuntu1) ...
Processing triggers for man-db (2.8.3-2) ...

Works fine.

3. Investigating strace also shows that the lock-frontend is acquired once, and not released until the end.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Julian Andres Klode (juliank) wrote :

FTR, with the locking with statement, without the SRU, it fails, as expected (both in bionic and xenial), e.g.:

dpkg: error: dpkg status database is locked by another process

E:Sub-process /usr/bin/dpkg returned an error code (2)
Traceback (most recent call last):
  File "/home/jak/Projects/Ubuntu/Scratch/foo.py", line 11, in <module>
    c.commit()
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 610, in commit
    raise SystemError("installArchives() failed")
SystemError: installArchives() failed

Revision history for this message
Julian Andres Klode (juliank) wrote :
Download full text (3.4 KiB)

Same for xenial (1.1.0~beta1ubuntu0.16.04.3).

Also, strace logs look like this, in case we want to see how we can be sure it's kept locked all the time:

[...]
open("/var/lib/dpkg/lock-frontend", O_RDWR|O_CREAT|O_NOFOLLOW, 0640) = 39
fcntl(39, F_SETFD, FD_CLOEXEC) = 0
fcntl(39, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
open("/var/lib/dpkg/lock", O_RDWR|O_CREAT|O_NOFOLLOW, 0640) = 40
fcntl(40, F_SETFD, FD_CLOEXEC) = 0
fcntl(40, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
open("/var/lib/dpkg/updates/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 41
close(41) = 0
close(41) = 0
close(41) = 0
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/lib/x86_64-linux-gnu/libnss_compat.so.2", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/lib/x86_64-linux-gnu/libnsl.so.1", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/lib/x86_64-linux-gnu/libnss_nis.so.2", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
close(41) = 0
close(41) = 0
open("/etc/group", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/group", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/var/lib/apt/lists/partial", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/group", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/var/cache/apt/archives/partial", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/etc/group", O_RDONLY|O_CLOEXEC) = 41
close(41) = 0
open("/var/cache/apt/archives/partial", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 41
close(41) = 0
open("/var/cache/apt/archives/lock", O_RDWR|O_CREAT|O_NOFOLLOW, 0640) = 41
fcntl(41, F_SETFD, FD_CLOEXEC) = 0
fcntl(41, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 42
close(42) = 0
open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 42
close(42) = 0
close(40) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=6098, si_uid=0, si_...

Read more...

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for python-apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-apt - 1.6.3

---------------
python-apt (1.6.3) bionic; urgency=medium

  * Frontend locking and related locking improvements (LP: #1795407)
    - apt.Cache: Keep / Re-establish the system lock in commit()
    - apt.Cache: Keep archive locked during commit()/ in fetch_archives()
    - apt.Cache: Reinstate locks in a finally / run dpkg inside try
    - Introduce frontend locking
    - Convert apt.Cache.commit and apt_pkg.DepCache.commit to FE lock
  * Other changes to make that work:
    - Cherry-pick apt_pkg.Error type hint from 1.7
    - travis CI: bionic only; stretch does not have FE locking nor PPA
  * As always, updated mirror lists

 -- Julian Andres Klode <email address hidden> Mon, 01 Oct 2018 16:00:14 +0200

Changed in python-apt (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Alexey Shchukin (alex-v-schukin) wrote :

Hello! Is there any change this fix will be backported to ubuntu/trusty?

Revision history for this message
Julian Andres Klode (juliank) wrote :

We don't plan to. Trusty does not run unattended-upgrades by default, so there's no strong need for frontend locking in there.

Revision history for this message
Alexey Shchukin (alex-v-schukin) wrote :

Got it. Thanks for the info.

description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

@juliank is this upload still relevant and good for release? Last time it was 'blocked' on the unattended-upgrades SRU - and that one has landed in xenial-updates already. Can we release it now, or does it need a re-upload?

Revision history for this message
Julian Andres Klode (juliank) wrote :

@sil2100 I think there is a regression in the other locking fixes, as reported in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922416 - but that said, we have that in every other release too, and I'll upload SRUs for that later.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Tracking the regression in bug 1826870

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted python-apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.1.0~beta1ubuntu0.16.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-xenial
removed: verification-done verification-done-xenial
Revision history for this message
Julian Andres Klode (juliank) wrote :

without SystemLock being held works fine:

root@xx:~# strace -o $PWD/log -e openat,open,close,fcntl python3 $PWD/foo.py
(Reading database ... 25835 files and directories currently installed.)
Removing aptitude-common (0.7.4-2ubuntu2) ...
COMMAND PID TYPE SIZE MODE M START END PATH
iscsid 432 POSIX 4B WRITE 0 0 0 /run/iscsid.pid
atd 421 POSIX 4B WRITE 0 0 0 /run/atd.pid
cron 418 FLOCK 4B WRITE 0 0 0 /run/crond.pid
python3 1207 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock-frontend
python3 1207 POSIX 0B WRITE 0 0 0 /var/cache/apt/archives/lock
dpkg 1215 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock
Processing triggers for man-db (2.7.5-1) ...

with SystemLock held works fine:
# strace -o $PWD/log.locked -e openat,open,close,fcntl python3 $PWD/foo.py
(Reading database ... 25835 files and directories currently installed.)
Removing aptitude-common (0.7.4-2ubuntu2) ...
COMMAND PID TYPE SIZE MODE M START END PATH
iscsid 432 POSIX 4B WRITE 0 0 0 /run/iscsid.pid
atd 421 POSIX 4B WRITE 0 0 0 /run/atd.pid
cron 418 FLOCK 4B WRITE 0 0 0 /run/crond.pid
dpkg 2285 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock
python3 2277 POSIX 0B WRITE 0 0 0 /var/lib/dpkg/lock-frontend
python3 2277 POSIX 0B WRITE 0 0 0 /var/cache/apt/archives/lock
Processing triggers for man-db (2.7.5-1) ...

In the strace log, we can see that the frontend lock is closed after the dpkg runs (after we have relocked the lock).

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Julian Andres Klode (juliank) wrote :

This was for xenial 1.1.0~beta1ubuntu0.16.04.4

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-apt - 1.1.0~beta1ubuntu0.16.04.4

---------------
python-apt (1.1.0~beta1ubuntu0.16.04.4) xenial; urgency=medium

  * apt.Cache: Fix (un)locking of archives (Closes: #922416) (LP: #1826870)
  * apt.Cache: Use explicit, more safe locking in update()
  * Update mirror lists

python-apt (1.1.0~beta1ubuntu0.16.04.3) xenial; urgency=medium

  * Frontend locking and related locking improvements (LP: #1795407)
    - apt.Cache: Keep / Re-establish the system lock in commit()
    - apt.Cache: Keep archive locked during commit()/ in fetch_archives()
    - apt.Cache: Reinstate locks in a finally / run dpkg inside try
    - Introduce frontend locking
    - Convert apt.Cache.commit and apt_pkg.DepCache.commit to FE lock
  * Other changes to make that work:
    - travis CI: enable PPA for new apt versions
  * As always, updated mirror lists

 -- Julian Andres Klode <email address hidden> Mon, 29 Apr 2019 14:11:32 +0200

Changed in python-apt (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.