apport python exception for python versions which python-apt is not built on

[Bart Goeman]

[Impact]
apport's python crash handler runs for every installed version of python e.g. on Ubuntu 18.04 LTS it will run for crashes with python3.6 or python3.7 as an interpreter. However, python apt modules are only available for the the supported version of python so the crash handler generates an exception which is annoying.

[Test Case for 18.04]
1) Install the non-default version of python3 e.g. python3.7 on Ubuntu 18.04 LTS.
2) Run 'python3.7 -c 'print hello' and observe a Traceback re apt_pkg not being found.

With the version of python3-apport from -proposed you'll not receive the Traceback from step # 2.

[Test Case for 20.04]
1) Install python3.7 and python3.8 on an Ubuntu 19.10 system
2) Upgrade from 19.10 to 20.04 LTS and keep python3.7 installed
3) Run 'python3.7 -c 'print hello' and observe a Traceback re apt_pkg not being found.

With the version of python3-apport from -proposed you'll not receive the Traceback from step # 2

[Regression Potential]
Its possible the crash handler change is incorrect so it should be confirmed that we still get python crashes about the system version of python. One way to do this is to run 'apport-cli coreutils < /dev/null'.

[Original Description]
it seems apport installs an exception hook whenever running python3.
This hook is apparently installed for python3.7 too.
This exception handler has a dependency on apt_pkg, which is a binary that is not compatible with python3.7 it seems (you can't import it, see below)

As a result, whenever I run a python3.7 script and run into an exception (which happens often when writing code), I get an additional exception in the exception handler + a copy of the original exception, so three stacktraces in total.
A solution would be to only install the exception hook for the system python i.e. python3.6

$ lsb_release -rd
Description: Ubuntu 18.04 LTS
Release: 18.04

apport: 2.20.9-0ubuntu7
python-apt: 1.6.0
python3.7: 3.7.0~b3-1

Demo with a trivial error:
$ python3 -c 'print hello'
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

$ python3.7 -c 'print hello'
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

Python 3.6.5 (default, Apr 1 2018, 05:46:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.excepthook
<function apport_excepthook at 0x7fdf2b99a620>
>>> import apt_pkg
>>> apt_pkg.__file__
'/usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so'
>>>

Python 3.7.0b3 (default, Mar 30 2018, 04:35:22)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.excepthook
<function apport_excepthook at 0x7f29a6eebea0>
>> import apt_pkg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'apt_pkg'

[Julian Andres Klode (juliank)]

python 3.7 is not a supported version, hence modules are not built for it.

[Julian Andres Klode (juliank)]

So apport needs to catch the exception and do nothing / not register an excepthook on unsupported python versions.

[Bart Goeman (bgoeman)]

well my reasoning was slightly different from supported/unsupported
i am not familiar with the exact scope of the apport project,
but I assume the goal is to collect info on ubuntu packages, not any unrelated code,
and since afaik all packages one one release all use the same (system) python version( i.e. 3.6 for bionic),
-> code running under any other python is of no interest anyhow
-> it only makes sense to install the apport exception hook for that python version,

[Stephen Croll (stephen-d-croll)]

As a workaround, I removed the 3.7 sitecustomize.py files:

  squirrel:~ 0:7> sudo rm /etc/python3.7/sitecustomize.py /usr/lib/python3.7/sitecustomize.py

The above files were apparently installed by libpython3.7-minimal:

  squirrel:~ 0:8> dpkg -S /etc/python3.7/sitecustomize.py /usr/lib/python3.7/sitecustomize.py
  libpython3.7-minimal:amd64: /etc/python3.7/sitecustomize.py
  libpython3.7-minimal:amd64: /usr/lib/python3.7/sitecustomize.py

[Jeff Larson (jmlarson)]

Any update on this issue?

[Calvin Cheng (calvincheng)]

Just ran into this same problem.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic

$ python3 --version
Python 3.7.3

$ apt-cache policy python3.7
python3.7:
  Installed: 3.7.3-2~18.04.1
  Candidate: 3.7.3-2~18.04.1
  Version table:
 *** 3.7.3-2~18.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
        100 /var/lib/dpkg/status
     3.7.0~b3-1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu34

---------------
apport (2.20.11-0ubuntu34) groovy; urgency=medium

  * apport_python_hook.py: if python apt modules are not built for the python
    version then do capture the crash. (LP: #1774843)

 -- Brian Murray <email address hidden> Wed, 13 May 2020 16:17:11 -0700

[Łukasz Zemczak (sil2100)]

Seeing that focal is an LTS, don't you think that it might still useful to have this 'fix' for that series? Currently we only have python3.8, but I assume we might backport newer python versions in the future. Wouldn't that make people potentially affected by this bug then?

I'll accept it as is, but let's reconsider pushing it to focal as well.

[Łukasz Zemczak (sil2100)]

Hello Bart, or anyone else affected,

Accepted apport into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu8.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Bart, or anyone else affected,

Accepted apport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.9-0ubuntu7.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apport (2.20.9-0ubuntu7.15) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.15 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#apport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Brian Murray (brian-murray)]

Verified on Ubuntu 18.04 LTS:

bdmurray@clean-bionic-amd64:~$ apt-cache policy python3-apport
python3-apport:
  Installed: 2.20.9-0ubuntu7.15
  Candidate: 2.20.9-0ubuntu7.15
  Version table:
 *** 2.20.9-0ubuntu7.15 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     2.20.9-0ubuntu7.14 500
        500 http://192.168.10.7/ubuntu bionic-updates/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-updates/main i386 Packages
        500 http://192.168.10.7/ubuntu bionic-security/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-security/main i386 Packages
     2.20.9-0ubuntu7 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic/main i386 Packages
bdmurray@clean-bionic-amd64:~$ python3.7 -c 'print hello'
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

[Brian Murray (brian-murray)]

Verified on Ubuntu 19.10:

bdmurray@clean-bionic-amd64:~$ apt-cache policy python3-apport
python3-apport:
  Installed: 2.20.9-0ubuntu7.15
  Candidate: 2.20.9-0ubuntu7.15
  Version table:
 *** 2.20.9-0ubuntu7.15 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     2.20.9-0ubuntu7.14 500
        500 http://192.168.10.7/ubuntu bionic-updates/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-updates/main i386 Packages
        500 http://192.168.10.7/ubuntu bionic-security/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-security/main i386 Packages
     2.20.9-0ubuntu7 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic/main i386 Packages
bdmurray@clean-bionic-amd64:~$ python3.7 -c 'print hello'
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

and the regression test passes:

bdmurray@clean-bionic-amd64:~$ apt-cache policy python3-apport
python3-apport:
  Installed: 2.20.9-0ubuntu7.15
  Candidate: 2.20.9-0ubuntu7.15
  Version table:
 *** 2.20.9-0ubuntu7.15 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     2.20.9-0ubuntu7.14 500
        500 http://192.168.10.7/ubuntu bionic-updates/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-updates/main i386 Packages
        500 http://192.168.10.7/ubuntu bionic-security/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-security/main i386 Packages
     2.20.9-0ubuntu7 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic/main i386 Packages
bdmurray@clean-bionic-amd64:~$ python3.7 -c 'print hello'
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

[Brian Murray (brian-murray)]

Hmm, that was a bad paste here is the regression test for 19.10:

bdmurray@clean-eoan-amd64:~$ apport-cli coreutils < /dev/null

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.....

*** Send problem report to the developers?

After the problem report has been sent, please fill out the form in the
automatically opened web browser.

What would you like to do? Your options are:
  S: Send report (1.6 KB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  I: Cancel and ignore future crashes of this program version
  C: Cancel
Please choose (S/V/K/I/C): Traceback (most recent call last):
  File "/usr/bin/apport-cli", line 387, in <module>
    if not app.run_argv():
  File "/usr/lib/python3/dist-packages/apport/ui.py", line 711, in run_argv
    return self.run_report_bug()
  File "/usr/lib/python3/dist-packages/apport/ui.py", line 550, in run_report_bug
    response = self.ui_present_report_details(allowed_to_report)
  File "/usr/bin/apport-cli", line 213, in ui_present_report_details
    response = dialog.run()
  File "/usr/bin/apport-cli", line 101, in run
    multi_char)
  File "/usr/bin/apport-cli", line 49, in raw_input_char
    saved_attributes = termios.tcgetattr(file)
termios.error: (25, 'Inappropriate ioctl for device')

And for Ubuntu 18.04 LTS:

bdmurray@clean-bionic-amd64:~$ apport-cli coreutils < /dev/null

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.....

*** Send problem report to the developers?

After the problem report has been sent, please fill out the form in the
automatically opened web browser.

What would you like to do? Your options are:
  S: Send report (1.6 KB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  I: Cancel and ignore future crashes of this program version
  C: Cancel
Please choose (S/V/K/I/C): Traceback (most recent call last):
  File "/usr/bin/apport-cli", line 387, in <module>
    if not app.run_argv():
  File "/usr/lib/python3/dist-packages/apport/ui.py", line 693, in run_argv
    return self.run_report_bug()
  File "/usr/lib/python3/dist-packages/apport/ui.py", line 532, in run_report_bug
    response = self.ui_present_report_details(allowed_to_report)
  File "/usr/bin/apport-cli", line 213, in ui_present_report_details
    response = dialog.run()
  File "/usr/bin/apport-cli", line 101, in run
    multi_char)
  File "/usr/bin/apport-cli", line 49, in raw_input_char
    saved_attributes = termios.tcgetattr(file)
termios.error: (25, 'Inappropriate ioctl for device')

[Bart Goeman (bgoeman)]

Verified on Ubuntu 19.10
on 19.10 python3-apt is built for pyton3.7, the system python for 19.10
so i tested against python3.8 from universe.

bart@DH170:~$ apt-cache policy python3-apport
python3-apport:
  Geïnstalleerd: 2.20.11-0ubuntu8.9
  Kandidaat: 2.20.11-0ubuntu8.9
  Versietabel:
 *** 2.20.11-0ubuntu8.9 400
        400 http://nl.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        400 http://nl.archive.ubuntu.com/ubuntu eoan-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     2.20.11-0ubuntu8.8 500
        500 http://nl.archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
        500 http://nl.archive.ubuntu.com/ubuntu eoan-updates/main i386 Packages
        500 http://nl.archive.ubuntu.com/ubuntu eoan-security/main amd64 Packages
        500 http://nl.archive.ubuntu.com/ubuntu eoan-security/main i386 Packages
     2.20.11-0ubuntu8 500
        500 http://nl.archive.ubuntu.com/ubuntu eoan/main amd64 Packages
        500 http://nl.archive.ubuntu.com/ubuntu eoan/main i386 Packages

bart@DH170:~$ python3.7 -c 'import apt_pkg'
bart@DH170:~$ python3.8 -c 'import apt_pkg'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'apt_pkg'
bart@DH170:~$ python3.8 -c 'print hello'
  File "<string>", line 1
    print hello
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

[Bart Goeman (bgoeman)]

Łukasz's remark on focal: +1

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.9-0ubuntu7.15

---------------
apport (2.20.9-0ubuntu7.15) bionic; urgency=medium

  * apport_python_hook.py: if python apt modules are not built for the python
    version then do capture the crash. (LP: #1774843)

 -- Brian Murray <email address hidden> Wed, 13 May 2020 15:12:14 -0700

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for apport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu8.9

---------------
apport (2.20.11-0ubuntu8.9) eoan; urgency=medium

  * apport_python_hook.py: if python apt modules are not built for the python
    version then do capture the crash. (LP: #1774843)

 -- Brian Murray <email address hidden> Wed, 13 May 2020 14:09:52 -0700

[Łukasz Zemczak (sil2100)]

Hello Bart, or anyone else affected,

Accepted apport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu27.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I successfully performed the verification on Ubuntu 20.04 LTS.

Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 python3-apport all 2.20.11-0ubuntu27.3 [83.8 kB]
Fetched 83.8 kB in 1s (117 kB/s)
(Reading database ... 307919 files and directories currently installed.)
Preparing to unpack .../python3-apport_2.20.11-0ubuntu27.3_all.deb ...
Unpacking python3-apport (2.20.11-0ubuntu27.3) over (2.20.11-0ubuntu27.2) ...
Setting up python3-apport (2.20.11-0ubuntu27.3) ...
bdmurray@clean-bionic-amd64:~$ python3.7 -c 'print hello'
  File "<string>", line 1
    print hello
              ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(hello)?

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu27.3

---------------
apport (2.20.11-0ubuntu27.3) focal; urgency=medium

  * apport_python_hook.py: if python apt modules are not built for the python
    version then do capture the crash. (LP: #1774843)
  * apport/report.py: If the user is not a part of any system groups then
    set UserGroups to 'N/A'. (LP: #1427600)

 -- Brian Murray <email address hidden> Mon, 01 Jun 2020 09:44:37 -0700