Aptdaemon crashes when there are any non-PEP-440 packages on the system due to setuptools 66.x.x

[Eugene Crosser]

[ Impact ]

"Software Updater" stopped working, it shows message "You stopped the check for updates".

update-manager.desktop[22442]: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying

Analysis shows that this is the result of aptdaemon crash when it tries to load plugins, and pkg_resources module refuses to parse non PEP-440 compliant Python package versions. There are many such python packages, one of them "devscripts", as shown in this log, but if I remove devscripts, it crashes on other packages.

org.debian.apt[22501]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2679, in parsed_version
org.debian.apt[22501]: self._parsed_version = parse_version(self.version)
org.debian.apt[22501]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
org.debian.apt[22501]: File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
org.debian.apt[22501]: raise InvalidVersion(f"Invalid version: '{version}'")
org.debian.apt[22501]: pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '2.22.2ubuntu5'
org.debian.apt[22501]: (package: devscripts)

There is an upstream ticket against setuptools: https://github.com/pypa/setuptools/issues/3772

[ Test Plan ]

1. Perform all required updates using Software Updater.
2. 'sudo apt install rmlint-gui'
3. Reboot (without reboot the bug did not reproduce)
4. Launch Software Updater: 'update-manager'
5. It will crash, see logs (08:38:56) related to apt daemon.
6. Upgrade the python3-pkg-resources
7. reboot
8. Launch Software Updater
9. Success

[ Where problems could occur ]

Building Python projects could start failing since the code is part of setuptools.

[ Other Info ]

There is a main ticket for fixing those non-PEP-440 packages: bug #1991606

ProblemType: Bug
DistroRelease: Ubuntu 23.04
Package: aptdaemon 1.1.1+bzr982-0ubuntu42
ProcVersionSignature: Ubuntu 5.19.0-29.30-generic 5.19.17
Uname: Linux 5.19.0-29-generic x86_64
ApportVersion: 2.24.0-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Fri Feb 24 00:16:28 2023
PackageArchitecture: all
SourcePackage: aptdaemon
UpgradeStatus: No upgrade log present (probably fresh install)

[Amberelle Mason (amberella-777)]

I opened bug #2008898 which may be a duplicate.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Amberelle Mason (amberella-777)]

Is this still an issue? I don't see it happening anymore.

[Eugene Crosser (crosser)]

@amberella-777 that is probably because you do not have any python packages whose version pkg_resources considers "invalid". In my case, it currently stumbles on `syncthing` package (from a third party repo)

Apr 11 20:01:01 pccross org.debian.apt[6869]: 20:01:01 AptDaemon [INFO]: Initializing daemon
Apr 11 20:01:01 pccross org.debian.apt[6869]: /usr/lib/python3/dist-packages/aptdaemon/worker/pkworker.py:35: PyGIWarning: PackageKitGlib was imported without specifying a version first. Use gi.require_version('PackageKitGlib', '1.0') before import to ensure that the right version gets loaded.
Apr 11 20:01:01 pccross org.debian.apt[6869]: from gi.repository import PackageKitGlib as pk
Apr 11 20:01:01 pccross org.debian.apt[6869]: Traceback (most recent call last):
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/aptdaemon/core.py", line 1462, in __init__
Apr 11 20:01:01 pccross org.debian.apt[6869]: from .worker.pkworker import AptPackageKitWorker
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/aptdaemon/worker/pkworker.py", line 43, in <module>
Apr 11 20:01:01 pccross org.debian.apt[6869]: from ..pkutils import (bitfield_add, bitfield_remove, bitfield_summarize,
Apr 11 20:01:01 pccross org.debian.apt[6869]: ModuleNotFoundError: No module named 'aptdaemon.pkutils'
Apr 11 20:01:01 pccross org.debian.apt[6869]: During handling of the above exception, another exception occurred:
Apr 11 20:01:01 pccross org.debian.apt[6869]: Traceback (most recent call last):
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/sbin/aptd", line 39, in <module>
Apr 11 20:01:01 pccross org.debian.apt[6869]: aptdaemon.core.main()
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/aptdaemon/core.py", line 2193, in main
Apr 11 20:01:01 pccross org.debian.apt[6869]: daemon = AptDaemon(options, bus=bus)
Apr 11 20:01:01 pccross org.debian.apt[6869]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/aptdaemon/core.py", line 1466, in __init__
Apr 11 20:01:01 pccross org.debian.apt[6869]: self.worker = AptWorker(options.chroot, load_plugins)
Apr 11 20:01:01 pccross org.debian.apt[6869]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/aptdaemon/worker/aptworker.py", line 205, in __init__
Apr 11 20:01:01 pccross org.debian.apt[6869]: self._load_plugins(["modify_cache_after", "modify_cache_before",
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/aptdaemon/worker/__init__.py", line 191, in _load_plugins
Apr 11 20:01:01 pccross org.debian.apt[6869]: dists, errors = pkg_resources.working_set.find_plugins(env)
Apr 11 20:01:01 pccross org.debian.apt[6869]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 11 20:01:01 pccross org.debian.apt[6869]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 907, in find_plugins
Apr 11 20:01:01 pccross org.debian.apt[6869]: env = Environment(self.entries)
Apr ...

[Amberelle Mason (amberella-777)]

@crosser thanks for the explanation. I'm running SyncThing in a snap.
https://gitlab.com/arubislander/syncthing-snap

[Eugene Crosser (crosser)]

For reference, I've submitted a pull request to syncthing-gtk to make their version PEP-440 compliant https://github.com/kozec/syncthing-gtk/pull/600.

I still think that it is wrong if any unrelated python package that is not fully standards compliant breaks the process of upgrade for the whole system.

[Dan Andreșan (danyer)]

In my case, rmlint-gui package was the culprit. Removed it and now I can use Software Updater (no more aptd crashes).

[Benjamin Drung (bdrung)]

There is a main ticket for fixing those non-PEP-440 packages: bug #1991606

[Robie Basak (racb)]

This bug was cited as justification for bug 1991606, so to help with understanding that, please could someone post steps to reproduce on a clean Ubuntu installation? Which Ubuntu releases are affected?

In particular, where did the affected /usr/lib/python3/dist-packages/pkg_resources/* come from? Is that from a package shipped by Ubuntu, and in which case, what versions are affected?

[Dan Andreșan (danyer)]

Interesting thing, adding back the rmlint-gui package (which was identified like one of the culprits, see #8) and I cannot reproduce the issue. Maybe it is gone?
Later Edit: after a reboot, the problem appears once more. I will create a new comment with steps to reproduce.

[Dan Andreșan (danyer)]

Steps to reproduce:

1. Install Ubuntu 23.04
2. Perform all required updates using Software Updater.
3. 'sudo apt install rmlint-gui'
4. Reboot (without reboot the bug did not reproduce)
5. Launch Software Updater
6. It will crash, see logs (08:38:56) related to apt daemon.
7. sudo apt remove rmlint-gui
8. Launch Software Updater
9. Success, see logs (08:40:20)

rmlint-gui is comming from universe repo, this is true, but the Software Updater should not crash.

[Eugene Crosser (crosser)]

@bdrung:

> There is a main ticket for fixing those non-PEP-440 packages: bug #1991606

this is not the point! Obviously non-PEP-440 compliant packages _in the distro_ must be fixed, independently of broken aptdaemon.

The point is that if somebody includes a third party repo, or installs a not-from-repo python package system-wide, and that package happens to be non-compliant, _it breaks distro upgrade process_. In my pretty firm opinion, it should no be this way.

I understand and agree with the wish to put more order in python packaging. Setuptools refuse to create a non-compliant package? That would be totally fair. But in this case, _uses_ who have no power to fix the problem, and may not even know what's going on, are taken hostage.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Norman Rieß (weuxel)]

Fixed for me in Mantic.

[Benjamin Drung (bdrung)]

The trigger is (still present in Ubuntu 23.10):

```
$ grep -r 2.9.0.Odd.Olm /usr/lib/python3*
/usr/lib/python3/dist-packages/Shredder-2.9.0.Odd.Olm.egg-info:Version: 2.9.0 Odd Olm
$ dpkg -S /usr/lib/python3/dist-packages/Shredder-2.9.0.Odd.Olm.egg-info
rmlint-gui: /usr/lib/python3/dist-packages/Shredder-2.9.0.Odd.Olm.egg-info
```

I can reproduce the behavior on Ubuntu 23.04 but not on 23.10 any more. So one component fixes it, but rmlist-gui and aptdaemon has not changed between.

[Benjamin Drung (bdrung)]

Updating python3-pkg-resources from 66.1.1-1 to 68.1.2-2 solved the failure. Following PR is probably the solution: https://github.com/pypa/setuptools/pull/3839

[Benjamin Drung (bdrung)]

Uploaded 66.1.1-1ubuntu0.1 and prepared the SRU template.

[Timo Aaltonen (tjaalton)]

Hello Eugene, or anyone else affected,

Accepted setuptools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/setuptools/66.1.1-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Knickers Brown (metta-crawler)]

Did you SRU for me? If so can I buy you a keto candy bar? Won't make you fat or give you calories. Everybody says sugar is bad, keto is sugar free so it's not bad, unless you count the bad way it tastes.

Sorry, I have a little puppy dog in my head and it get so exited sometimes.

Be well,
Whoever I am today.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted setuptools (66.1.1-1ubuntu0.1) for lunar have finished running.
The following regressions have been reported in tests triggered by the package:

patroni/3.0.1-1 (amd64, arm64, armhf, ppc64el, s390x)
pyflakes/2.5.0-1 (arm64, armhf, ppc64el, s390x)
python-can/4.1.0-1 (s390x)
python-libtmux/0.21.0-1 (amd64, armhf, ppc64el)
radicale/3.0.6-4ubuntu2 (arm64, ppc64el)
rdflib/6.1.1-1 (amd64, arm64, armhf, ppc64el, s390x)
ros-catkin-tools/0.9.2+ds-1 (armhf)
snakemake/7.21.0-1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/lunar/update_excuses.html#setuptools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Benjamin Drung (bdrung)]

Yes, I did SRU the fix for you and all other affected people. No need for candy bar, but I appreciate the positive feedback. :)

[Benjamin Drung (bdrung)]

Verified in a lunar VM: Installed rmlint-gui, rebooted, trigger crash, installed python3-pkg-resources=66.1.1-1ubuntu0.1 and the crash is gone.

[Eugene Crosser (crosser)]

🎉 @bdrung

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for setuptools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package setuptools - 66.1.1-1ubuntu0.1

---------------
setuptools (66.1.1-1ubuntu0.1) lunar; urgency=medium

  * Make pkg_resources more forgiving of non-compliant versions (LP: #2008430)

 -- Benjamin Drung <email address hidden> Wed, 25 Oct 2023 12:01:45 +0200