Object of different cache passed as argument to apt_pkg.DepCache method

[errors.ubuntu.com bug bridge]

[Impact]
python-apt 1.6 raises an exception when objects of an old cache are passed to a apt_pkg.DepCache methods for a different cache. Prior to that, those would either segfault, succeed, or silently operate on a different object, as they use package/version ids, and e.g. two different packages in the old and new cache might have the same id.

With 1.6.1, we introduce a remapping algorithm that remaps objects of apt.Cache() when calling apt.Cache.open(), allowing old objects to be used after reopening, as long as they exist in the new cache. If they don't exist in the new cache, apt_pkg.CacheMismatchError will be raised from the apt_pkg layer.

[Test case]
import apt
c=apt.Cache()
p=c["apt"]
c.open()
p.mark_install()

[Regression potential]
Could be remapping to wrong items which would cause us to install a wrong version, for example. Compared to pre-bionic, bionic is a regression already, though, and any regression caused here is less important than
what we have now.

[Original bug report]
The Ubuntu Error Tracker has been receiving reports about a problem regarding update-manager. This problem was most recently seen with package version 1:18.04.11, the problem page at https://errors.ubuntu.com/problem/e6ff7b5c385c512b7933497ad895c8a19ed063b2 contains more details, including versions of packages affected, stacktrace or traceback, and individual crash reports.
If you do not have access to the Ubuntu Error Tracker and are a software developer, you can request it at http://forms.canonical.com/reports/.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.7.0~alpha0~ubuntu3

---------------
python-apt (1.7.0~alpha0~ubuntu3) cosmic; urgency=medium

  * apt.Cache: Fix error in apt.Package caching
  * apt: Fix typing errors and use mypy --strict
  * Correctly handle missing candidate in Package.get_changelog
  * apt.Cache: Remap objects when reopening cache (LP: #1773316)
  * apt_pkg.DepCache: Raise CacheMismatchError if argument belongs to diff. cache
  * apt.debfile: Avoid exception in replaces_real_pkg for no-candidate packages
  * apt.debfile.DebPackage: Handle no-candidate package in _maybe_append_multiarch_suffix
  * utils/get_debian_mirrors.py: Get data from salsa

 -- Julian Andres Klode <email address hidden> Wed, 06 Jun 2018 16:09:32 -0700

[Steve Langasek (vorlon)]

Hello errors.ubuntu.com, or anyone else affected,

Accepted python-apt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-apt/1.6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Julian Andres Klode (juliank)]

Fix works fine:

root@bionic:~# python3 -c "import apt; c=apt.Cache(); p=c['aptitude']; c.open(); p.mark_install()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/apt/package.py", line 1469, in mark_install
    self._pcache._depcache.mark_install(self._pkg, auto_inst, from_user)
ValueError: Object of different cache passed as argument to apt_pkg.DepCache method
root@bionic:~# dpkg -i python3-apt_1.6.1_amd64.deb
(Reading database ... 29845 files and directories currently installed.)
Preparing to unpack python3-apt_1.6.1_amd64.deb ...
Unpacking python3-apt (1.6.1) over (1.6.0) ...
Setting up python3-apt (1.6.1) ...
root@bionic:~# python3 -c "import apt; c=apt.Cache(); p=c['aptitude']; c.open(); p.mark_install()"

another check:

root@bionic:~# python3 -c "import apt; c=apt.Cache(); p=c['aptitude']; c.open(); p.mark_delete(); print(c.get_changes()); c.commit()"
[<Package: name:'aptitude' architecture='amd64' id:257>]
(Reading database ... 29852 files and directories currently installed.)
Removing aptitude (0.8.10-6ubuntu1) ...

[Julian Andres Klode (juliank)]

I also ran unattended-upgrades and aptdcon --system-upgrade, both of which performed normally.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for python-apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 1.6.1

---------------
python-apt (1.6.1) unstable; urgency=medium

  * apt: Fix typing errors to get CI pass again
  * apt.Cache: Remap objects when reopening cache (LP: #1773316)
  * apt_pkg.DepCache: Raise CacheMismatchError if argument belongs to diff. cache
  * Set branch to 1.6.y and adjust travis CI to run against stable releases
  * travis CI / Dockerfile: Pin mypy to 0.600 to prevent future changes breaking CI
  * utils/get_debian_mirrors.py: Get data from salsa

 -- Julian Andres Klode <email address hidden> Wed, 06 Jun 2018 15:14:59 -0700

[Julian Andres Klode (juliank)]

This update causes a regression that is tracked in bug 1780099 - notably the __hash__ of an apt.Package can change after opening, so if used in a set, or as a dict key, the package won't be found anymore. The regression is less worse than the previous state, but will be fixed shortly.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

JFTR, in case anyone is looking at this:

The fix for xenial and trusty has been merged into the fix for 1737441, and was fixed there, as the regression that manifested in this bug due to the fix for 1737441 never manifested in those releases, as there was no upload with just that fix.