systemd ProtectSystem/mount namespace makes apport fail (impact most of our default system services)

Bug #1870060 reported by Daniel van Vugt on 2020-04-01
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apport (Ubuntu)
Status tracked in Hirsute
Focal
High
Unassigned
Groovy
High
Unassigned
Hirsute
High
Brian Murray

Bug Description

[Impact]
apport is not creating crash files for multiple system services that use ProtectSystem since Ubuntu 20.04 LTS.

[Test Case]
On an Ubuntu desktop system perform the following step
1) sudo pkill -11 upowerd

With the current version of apport there will not be a crash file for upowerd in /var/crash/. With the version of apport in -proposed there will be a crash file for upowerd in /var/crash/.

[Regression Potential]
In the event that the changes to /usr/share/apport/apport are bad then apport itself would crash thereby preventing us from getting any crash reports. So in addition to killing upowerd we should also test a regular application like xeyes.

[Original Description]
bluetoothd never leaves dumps/crash files when it crashes

And it seems this is true for everyone. Only supplementary binaries' crash reports are shown:
https://errors.ubuntu.com/?release=Ubuntu%2020.04&package=bluez&period=year

Related branches

tags: added: champagne
Brian Murray (brian-murray) wrote :

There are bluetoothd crashes in the Error Tracker but the most recent release about which they were reported is 17.10.

Brian Murray (brian-murray) wrote :

Killing bluetoothd with signall 11 on Ubuntu 18.04 LTS I was able to create and submit a crash report to the Error Tracker.

https://errors.ubuntu.com/oops/88ed113a-7415-11ea-98e4-fa163e102db1

Sebastien Bacher (seb128) wrote :

It's due to the system bluetooth.service unit using ProtectSystem=full, commenting that line from the service and doing
$ sudo systemctl stop blueetooth.service
$ sudo systemctl daemon-reload
$ sudo systemctl start blueetooth.service

and sending the sig11 makes apport trigger

Note that systemd-coredump doesn't have this problem and register a backtrace in the journal even with ProtectSystem in use, which suggests it's an apport limitation

summary: - bluetoothd never leaves dumps/crash files when it crashes
+ systemd ProtectSystem makes apport fail
summary: - systemd ProtectSystem makes apport fail
+ systemd ProtectSystem makes apport fail (impacts bluez)
affects: whoopsie (Ubuntu) → apport (Ubuntu)
tags: added: rls-ff-incoming
summary: - systemd ProtectSystem makes apport fail (impacts bluez)
+ systemd ProtectSystem/mount namespace makes apport fail (impacts bluez)

Investigating a bit, it looks like apport bails out because the process hits this error
'host pid %s crashed in a separate mount namespace, ignoring'

systemd does use mount namespaces for its security features, apport shouldn't bail out in that context or we loose reporting for important system services (and we don't want to disable sandboxing to workaround the issue)

Changed in apport (Ubuntu):
importance: Undecided → High
Sebastien Bacher (seb128) wrote :

$ grep ProtectSystem /lib/systemd/system/*.service

In fact most system services in focal seem to fail triggering apport, tried on upower or bolt leads to the same error

summary: - systemd ProtectSystem/mount namespace makes apport fail (impacts bluez)
+ systemd ProtectSystem/mount namespace makes apport fail (impact most of
+ our default system services)
Changed in bluez (Ubuntu):
status: New → Invalid
tags: removed: rls-ff-incoming
Steve Langasek (vorlon) wrote :

Foundations is taking this for focal but it's at risk for the GA release.

If there are system services that are crashing, and this issue is getting in the way of the Desktop Team debugging those crashes, a workaround would be to disable the ProtectSystem rule in the systemd units (or instruct users how to do so locally). This is defense in depth, but is not a core part of our security model in Ubuntu that we should be unwilling to disable it for debugging.

Sebastien Bacher (seb128) wrote :

Thanks Steve, local debugging isn't the concern at this point (the lack of reporting on e.u.c is the main issue) but that's good to know still.

Daniel van Vugt (vanvugt) wrote :

I don't consider this bug "High" myself. It was just an interesting observation and isn't blocking me.

tags: removed: champagne
tags: added: id-5e8615c94049a02d98879c25

FYI - I'm facing a bug where it would have been really helpful to have this fixed for better reporting. Therefore giving this nudge by pinging here ...

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apport (Ubuntu Focal):
status: New → Confirmed
Changed in apport (Ubuntu):
status: New → Confirmed
Dan Streetman (ddstreet) wrote :

maybe it would be better to just switch to systemd-coredump for crash handling.

Sebastien Bacher (seb128) wrote :

@Dan, that could be a possible option going forward, from testing it doesn't have that limitation and it might have a more active upstream but probably not a change for a SRU so we would need to fix anyway for the LTS

tags: added: fr-27
Changed in apport (Ubuntu):
assignee: nobody → Brian Murray (brian-murray)
status: Confirmed → In Progress
description: updated
Changed in apport (Ubuntu Groovy):
status: New → Triaged
importance: Undecided → High
Changed in apport (Ubuntu Focal):
status: Confirmed → Triaged
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.11-0ubuntu51

---------------
apport (2.20.11-0ubuntu51) hirsute; urgency=medium

  * data/apport: Modify the check for whether or not a process is running in
    the same namespace so that crashes from processes running protected in the
    system.slice are considered as being from the same namespace. (LP: #1870060)
  * etc/apport/crashdb.conf: Enable Launchpad crash reports for 21.04.

 -- Brian Murray <email address hidden> Wed, 04 Nov 2020 13:40:41 -0800

Changed in apport (Ubuntu):
status: In Progress → Fix Released
Changed in bluez (Ubuntu Groovy):
status: New → Invalid
Sebastien Bacher (seb128) wrote :

Confirmed that with the update e.g upowerd reports are generated

Hello Daniel, or anyone else affected,

Accepted apport into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu50.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apport (Ubuntu Groovy):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-groovy
Łukasz Zemczak (sil2100) wrote :

Hello Daniel, or anyone else affected,

Accepted apport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu27.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apport (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed-focal
Brian Murray (brian-murray) wrote :

I had to reboot for the verification to pass, but it did.

bdmurray@clean-groovy-amd64:~$ apt list apport
Listing... Done
apport/groovy-proposed,groovy-proposed,now 2.20.11-0ubuntu50.2 all [installed,automatic]
N: There are 2 additional versions. Please use the '-a' switch to see them.
bdmurray@clean-groovy-amd64:~$ sudo pkill -11 upowerd
bdmurray@clean-groovy-amd64:~$ ls /var/crash/
_usr_libexec_upowerd.0.crash

Brian Murray (brian-murray) wrote :

I tested groovy again, but this time I installed apport and python3-apport in one step instead of two and it passed without needing to reboot.

The verification on Focal also passed with no rebooting required.

bdmurray@clean-focal-amd64:~$ apt list apport
Listing... Done
apport/focal-proposed,focal-proposed,now 2.20.11-0ubuntu27.13 all [installed]
N: There are 2 additional versions. Please use the '-a' switch to see them.
bdmurray@clean-focal-amd64:~$ sudo pkill -11 upowerd
bdmurray@clean-focal-amd64:~$ ls /var/crash/
_usr_lib_upower_upowerd.0.crash

tags: added: verification-done verification-done-focal verification-done-groovy
removed: verification-needed verification-needed-focal verification-needed-groovy
Mathew Hodson (mhodson) on 2020-11-23
no longer affects: bluez (Ubuntu)
no longer affects: bluez (Ubuntu Focal)
no longer affects: bluez (Ubuntu Hirsute)
no longer affects: bluez (Ubuntu Groovy)

The verification of the Stable Release Update for apport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.11-0ubuntu27.13

---------------
apport (2.20.11-0ubuntu27.13) focal; urgency=medium

  * data/apport: Modify the check for whether or not a process is running in
    the same namespace so that crashes from processes running protected in the
    system.slice are considered as being from the same namespace. (LP: #1870060)

 -- Brian Murray <email address hidden> Mon, 16 Nov 2020 14:40:03 -0800

Changed in apport (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.11-0ubuntu50.2

---------------
apport (2.20.11-0ubuntu50.2) groovy; urgency=medium

  * data/apport: Modify the check for whether or not a process is running in
    the same namespace so that crashes from processes running protected in the
    system.slice are considered as being from the same namespace. (LP: #1870060)

 -- Brian Murray <email address hidden> Mon, 16 Nov 2020 14:33:58 -0800

Changed in apport (Ubuntu Groovy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers