apport-gtk keeps prompting to report crashes in a loop

[Steve Langasek]

/var/crash has the following contents:

-rw-r----- 1 vorlon vorlon 107490 May 23 16:05 _usr_bin_gourmand.1000.crash
-rw-rw-r-- 1 vorlon vorlon 0 May 23 16:06 _usr_bin_gourmand.1000.upload
-rw------- 1 whoopsie whoopsie 5 May 23 16:07 _usr_bin_gourmand.1000.uploaded
-rw-r----- 1 vorlon vorlon 23647 May 23 13:57 _usr_bin_python-argcomplete-check-easy-install-script.1000.crash
-rw-rw-r-- 1 vorlon vorlon 0 May 23 13:59 _usr_bin_python-argcomplete-check-easy-install-script.1000.upload
-rw------- 1 whoopsie whoopsie 5 May 23 13:59 _usr_bin_python-argcomplete-check-easy-install-script.1000.uploaded

As seen, the .upload / .uploaded files were created within 2 minutes of each of the crashes.

But after submitting the crash report for /usr/bin/python-argcomplete-check-easy-install-script, I left my desktop for a while. I came back to find at least TWENTY dialogs prompting me to submit the same crash report.

The only way to stop the system from creating more prompts is to remove these files from disk.

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: apport 2.28.1-0ubuntu3
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
NonfreeKernelModules: zfs
ApportLog:

ApportVersion: 2.28.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Thu May 23 16:06:43 2024
InstallationDate: Installed on 2019-12-23 (1613 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
PackageArchitecture: all
SourcePackage: apport
UpgradeStatus: Upgraded to noble on 2024-05-22 (1 days ago)

[Benjamin Drung (bdrung)]

Can you provide the journal log from during that time frame? Did that application crash multiple times?

[Benjamin Drung (bdrung)]

Why is ApportLog empty? Did you remove the content there? Otherwise it means that /var/log/apport was empty (despite apport collecting crashes).

[Steve Langasek (vorlon)]

> Did you remove the content there?

I did not.

[Steve Langasek (vorlon)]

> Can you provide the journal log from during that time frame?

For what service? For whoopsie, I see:

May 23 13:58:13 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:58:13 homer whoopsie[147832]: [13:58:13] Using lock path: /var/lock/whoopsie/lock
May 23 13:58:13 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:58:13 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:58:13 homer whoopsie[147855]: [13:58:13] Using lock path: /var/lock/whoopsie/lock
May 23 13:58:13 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:58:20 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:58:20 homer whoopsie[148149]: [13:58:20] Using lock path: /var/lock/whoopsie/lock
May 23 13:58:20 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:58:44 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:58:44 homer whoopsie[148216]: [13:58:44] Using lock path: /var/lock/whoopsie/lock
May 23 13:58:44 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:58:48 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:58:48 homer whoopsie[148496]: [13:58:48] Using lock path: /var/lock/whoopsie/lock
May 23 13:58:48 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:59:11 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:59:11 homer whoopsie[148522]: [13:59:11] Using lock path: /var/lock/whoopsie/lock
May 23 13:59:11 homer whoopsie[148522]: [13:59:11] Parsing /var/crash/_usr_bin_python-argcomplete-check-easy-install-script.1000.crash.
May 23 13:59:11 homer whoopsie[148522]: [13:59:11] Unable to parse report (/var/crash/_usr_bin_python-argcomplete-check-easy-install-script.1000.crash): /var/crash/_usr_bin_python-argcomplete-check-easy-install-script.1000.crash could not be opened.
May 23 13:59:11 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:59:25 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:59:25 homer whoopsie[148594]: [13:59:25] Using lock path: /var/lock/whoopsie/lock
May 23 13:59:25 homer whoopsie[148594]: [13:59:25] Parsing /var/crash/_usr_bin_python-argcomplete-check-easy-install-script.1000.crash.
May 23 13:59:25 homer whoopsie[148594]: [13:59:25] Unable to parse report (/var/crash/_usr_bin_python-argcomplete-check-easy-install-script.1000.crash): /var/crash/_usr_bin_python-argcomplete-check-easy-install-script.1000.crash could not be opened.
May 23 13:59:25 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:59:25 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:59:25 homer whoopsie[148608]: [13:59:25] Using lock path: /var/lock/whoopsie/lock
May 23 13:59:25 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 13:59:29 homer systemd[1]: Started whoopsie.service - crash report submission.
May 23 13:59:29 homer whoopsie[148618]: [13:59:29] Using lock path: /var/lock/whoopsie/lock
May 23 13:59:29 homer systemd[1]: whoopsie.service: Deactivated successfully.
May 23 14:01:44 home...

[Simon Chopin (schopin)]

Steve, do you still have the crash file on your system? If so, could you give us its mod and owner?

[Simon Chopin (schopin)]

Interestingly, I see the same "could not be opened" patter in my recent logs, all for crashes within Python executables.

[Steve Langasek (vorlon)]

> Steve, do you still have the crash file on your system?

No, but I have no trouble generating more of them on demand.

> If so, could you give us its mod and owner?

-rw-r----- 1 vorlon vorlon 41580 Jun 3 17:51 /var/crash/_usr_bin_grep-merges.1000.crash

[Simon Chopin (schopin)]

I found the wild goose!

Your Python crashes are owned by vorlon:vorlon, which is why whoopsie cannot read them. I'm guessing that /var/crash doesn't have the 3777 permissions that would allow inheriting the whoopsie GID. My own system has 1777, whereas fresh containers seem to be correctly using 3777.

It seems to boil down to a race condition between the apport postinst and the whoopsie postinst, the latter correctly chmoding /var/crash to 3777 *if it creates it*. We should fix the apport postinst to use the proper permissions and fix existing installs.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Chopin (schopin)]

Note that I still don't understand why it loops, whereas on my system whoopsie will happily create the upload/uploaded files (with a NULL ID).

[Simon Chopin (schopin)]

Could we have the apport-autoreport.service journal logs as well?

I'm now realizing that the reason I don't get spammed could simply be that I configured my system to report everything.

[Steve Langasek (vorlon)]

$ journalctl --no-pager -lu apport-autoreport.service
-- No entries --
$

FTR:

             │ ├─app-gnome-update\x2dnotifier-14736.scope
             │ │ ├─ 14736 /usr/bin/update-notifier
             │ │ ├─1602383 /usr/lib/update-notifier/livepatch-notification
             │ │ ├─2268421 /usr/bin/python3 /usr/share/apport/apport-gtk
             │ │ ├─2268570 /usr/bin/python3 /usr/share/apport/apport-gtk
             │ │ └─2268868 /usr/bin/python3 /usr/share/apport/apport-gtk

[Steve Langasek (vorlon)]

$ ls -ld /var/crash/
drwxrwxrwt 2 root whoopsie 3 Jun 7 15:16 /var/crash/
$

[Steve Langasek (vorlon)]

> It seems to boil down to a race condition between the apport postinst
> and the whoopsie postinst, the latter correctly chmoding /var/crash
> to 3777 *if it creates it*.

My read of the postinst is that it correctly chmod's /var/crash regardless of whether it did the creating:

                mkdir -p -m 3777 /var/crash
                chmod g+s /var/crash
                chgrp whoopsie /var/crash

I've confirmed that 'mkdir -p -m' doesn't change the mode of an existing directory. And /var/crash has been on my system since 2019 when this system was installed, so it's not new here?

Running `dpkg-reconfigure whoopsie` (i.e., triggering a rerun of the postinst) is sufficient to fix the directory permissions.

But it must have been something else besides the apport postinst that dropped the g+s bit.

[Simon Chopin (schopin)]

Somehow, I missed the chmod line in that postinst.

My current hypothesis is that apport.service is to blame for dropping the g+s bit. It's the oneshot service that runs on non-container systems at startup to set up the kernel crash handler (was an init script before Lunar), and it forces the 1777 permissions. I just confirmed that it's 1777 on the live CD in a VM for Noble, so any Python crash there is basically ignored, even if the user clicks "Send".

That still doesn't tell us why it's looping on your system, though.

[Simon Chopin (schopin)]

I've finally tracked down the permission issue:

In Jammy, we still have an old-school init script that's invoked to set up the crash handler at the kernel level. As part of the script, there an unconditional `chmod 1777 /var/run`. However, surprisingly, `chmod 1777` on a 3777 file will result in a 3777 file (that's actually documented in the man page). To confirm, I straced the chmod execution, and the syscall is fed 3777, so we can't even blame this on the kernel ;).

In the Lunar cycle we moved away from the init file to use a proper service file instead (see https://github.com/canonical/apport/pull/38 ). As part of this move, the `chmod(1)` call was translated into a Python `os.chmod` call, which will do a straight syscall without the fancy logic of chmod(1), with the end result of whoopsie being unable to read Python crash files.

[Benjamin Drung (bdrung)]

Proposed changes for the apport.service: https://github.com/canonical/apport/pull/349

In addition the apport postinst should create and update the permission to 3777.