systemd ProtectSystem/mount namespace makes apport fail (impact most of our default system services)

[Daniel van Vugt]

[Impact]
apport is not creating crash files for multiple system services that use ProtectSystem since Ubuntu 20.04 LTS.

[Test Case]
On an Ubuntu desktop system perform the following step
1) sudo pkill -11 upowerd

With the current version of apport there will not be a crash file for upowerd in /var/crash/. With the version of apport in -proposed there will be a crash file for upowerd in /var/crash/.

[Regression Potential]
In the event that the changes to /usr/share/apport/apport are bad then apport itself would crash thereby preventing us from getting any crash reports. So in addition to killing upowerd we should also test a regular application like xeyes.

[Original Description]
bluetoothd never leaves dumps/crash files when it crashes

And it seems this is true for everyone. Only supplementary binaries' crash reports are shown:
https://errors.ubuntu.com/?release=Ubuntu%2020.04&package=bluez&period=year

[Brian Murray (brian-murray)]

There are bluetoothd crashes in the Error Tracker but the most recent release about which they were reported is 17.10.

[Brian Murray (brian-murray)]

Killing bluetoothd with signall 11 on Ubuntu 18.04 LTS I was able to create and submit a crash report to the Error Tracker.

https://errors.ubuntu.com/oops/88ed113a-7415-11ea-98e4-fa163e102db1

[Sebastien Bacher (seb128)]

It's due to the system bluetooth.service unit using ProtectSystem=full, commenting that line from the service and doing
$ sudo systemctl stop blueetooth.service
$ sudo systemctl daemon-reload
$ sudo systemctl start blueetooth.service

and sending the sig11 makes apport trigger

Note that systemd-coredump doesn't have this problem and register a backtrace in the journal even with ProtectSystem in use, which suggests it's an apport limitation

[Sebastien Bacher (seb128)]

Investigating a bit, it looks like apport bails out because the process hits this error
'host pid %s crashed in a separate mount namespace, ignoring'

systemd does use mount namespaces for its security features, apport shouldn't bail out in that context or we loose reporting for important system services (and we don't want to disable sandboxing to workaround the issue)

[Sebastien Bacher (seb128)]

$ grep ProtectSystem /lib/systemd/system/*.service

In fact most system services in focal seem to fail triggering apport, tried on upower or bolt leads to the same error

[Steve Langasek (vorlon)]

Foundations is taking this for focal but it's at risk for the GA release.

If there are system services that are crashing, and this issue is getting in the way of the Desktop Team debugging those crashes, a workaround would be to disable the ProtectSystem rule in the systemd units (or instruct users how to do so locally). This is defense in depth, but is not a core part of our security model in Ubuntu that we should be unwilling to disable it for debugging.

[Sebastien Bacher (seb128)]

Thanks Steve, local debugging isn't the concern at this point (the lack of reporting on e.u.c is the main issue) but that's good to know still.

[Daniel van Vugt (vanvugt)]

I don't consider this bug "High" myself. It was just an interesting observation and isn't blocking me.

[Christian Ehrhardt  (paelzer)]

FYI - I'm facing a bug where it would have been really helpful to have this fixed for better reporting. Therefore giving this nudge by pinging here ...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dan Streetman (ddstreet)]

maybe it would be better to just switch to systemd-coredump for crash handling.

[Sebastien Bacher (seb128)]

@Dan, that could be a possible option going forward, from testing it doesn't have that limitation and it might have a more active upstream but probably not a change for a SRU so we would need to fix anyway for the LTS

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu51

---------------
apport (2.20.11-0ubuntu51) hirsute; urgency=medium

  * data/apport: Modify the check for whether or not a process is running in
    the same namespace so that crashes from processes running protected in the
    system.slice are considered as being from the same namespace. (LP: #1870060)
  * etc/apport/crashdb.conf: Enable Launchpad crash reports for 21.04.

 -- Brian Murray <email address hidden> Wed, 04 Nov 2020 13:40:41 -0800

[Sebastien Bacher (seb128)]

Confirmed that with the update e.g upowerd reports are generated

[Łukasz Zemczak (sil2100)]

Hello Daniel, or anyone else affected,

Accepted apport into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu50.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Daniel, or anyone else affected,

Accepted apport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu27.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I had to reboot for the verification to pass, but it did.

bdmurray@clean-groovy-amd64:~$ apt list apport
Listing... Done
apport/groovy-proposed,groovy-proposed,now 2.20.11-0ubuntu50.2 all [installed,automatic]
N: There are 2 additional versions. Please use the '-a' switch to see them.
bdmurray@clean-groovy-amd64:~$ sudo pkill -11 upowerd
bdmurray@clean-groovy-amd64:~$ ls /var/crash/
_usr_libexec_upowerd.0.crash

[Brian Murray (brian-murray)]

I tested groovy again, but this time I installed apport and python3-apport in one step instead of two and it passed without needing to reboot.

The verification on Focal also passed with no rebooting required.

bdmurray@clean-focal-amd64:~$ apt list apport
Listing... Done
apport/focal-proposed,focal-proposed,now 2.20.11-0ubuntu27.13 all [installed]
N: There are 2 additional versions. Please use the '-a' switch to see them.
bdmurray@clean-focal-amd64:~$ sudo pkill -11 upowerd
bdmurray@clean-focal-amd64:~$ ls /var/crash/
_usr_lib_upower_upowerd.0.crash

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for apport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu27.13

---------------
apport (2.20.11-0ubuntu27.13) focal; urgency=medium

  * data/apport: Modify the check for whether or not a process is running in
    the same namespace so that crashes from processes running protected in the
    system.slice are considered as being from the same namespace. (LP: #1870060)

 -- Brian Murray <email address hidden> Mon, 16 Nov 2020 14:40:03 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.20.11-0ubuntu50.2

---------------
apport (2.20.11-0ubuntu50.2) groovy; urgency=medium

  * data/apport: Modify the check for whether or not a process is running in
    the same namespace so that crashes from processes running protected in the
    system.slice are considered as being from the same namespace. (LP: #1870060)

 -- Brian Murray <email address hidden> Mon, 16 Nov 2020 14:33:58 -0800