Ubuntu 20.04: opal-prd fails to start on 20.04

[bugproxy]

[Impact]
opal-prd (the daemon on a power system that listens for hw diagnostic system events at the /dev/opal-prd device) fails to start.
The reason is that opal-prd is not able to properly handle devtmpfs, mounted with noexec in /dev, which is the case on recent versions of systemd (like used in focal or newer)..
Currently such a mount prevents mapping HBRT image code region as 'rwx' from /dev.

[Fix]
This patch/commit attempts to work around the situation by copying HBRT image to a non mmapped memory region and sets mprotect rwx on it, allowing opal-prd to successfully execute the code region (as suggested here: https://github.com/open-power/skiboot/issues/258):
47005e8d4c9aeda5826c17c4a013cfbda1a3f2de 47005e8 "opal-prd: handle devtmpfs mounted with noexec"

[Test Case]
Since the opal-prd daemon must be running in the background as a separate process, the test is to:
- install the updated package that includes the patched opal-prd daemon (e.g. from the PPA mentioned below)
- double check the installed package version (dpkg -l) and maybe the opal-pd version that's in place (opal-prd --version)
- start opal-prd as daemon: 'service opal-prd start' (if not started automatically)
- verify the opal-prd status and check if it's running or not, by for example 'service opal-prd status'

[What could go wrong]
Things can go wrong in case the HBRT image copy is done wrong; in case it's accidentally copied to a wrong memory area (e.g. to an already mapped range, or erroneously calculated address/size), a seg. fault will happen and the system would core dump.
The mprotect code is pretty straight forward, but the fact that mprotect rwx is set on it, allows opal-prd to successfully execute the code region. It's not generally a perfect approach to map memory as RWX, but HBRT requires the ability to write into the image at runtime - and it got upstream accepted that way with skiboot v6.7.
The fix was released back in October and was pre-tested by the IBM Power team.
On top a patched Ubuntu package was build and shared in a PPA (see comment #1) and again successfully validated on focal and groovy.
__________

[Original Description]

== Comment: #0 - VASANT HEGDE <email address hidden> - 2020-11-23 23:23:22 ==
---Problem Description---
opal-prd fails to start on 20.04

Contact Information = Vasant hegde <email address hidden>

---uname output---
Ubuntu 20.04

Machine Type = All Power System

---Steps to Reproduce---
 opal-prd fails to start on 20.04

Userspace tool common name: opal-prd

The userspace tool has the following bit modes: 64bit

Userspace rpm: opal-prd

This is fixed in upstream by below commit. Please backport this patch to 20.04 LTS release. Also applicable for 20.10.

commit 47005e8d4c9aeda5826c17c4a013cfbda1a3f2de
Author: Georgy Yakovlev <email address hidden>
Date: Mon Oct 12 14:29:17 2020 -0700

    opal-prd: handle devtmpfs mounted with noexec

    On systems using recent versions of systemd /dev (devtmpfs) is mounted with
    noexec option. Such mount prevents mapping HBRT image code region as RWX
    from /dev. This commit, as suggested in github PR linked below, attempts to
    work around the situation by copying HBRT image to anon mmaped memory
    region and sets mprotect rwx on it, allowing opal-prd to sucessfully
    execute the code region.

    Having memory region set as RWX is not ideal for security, but fixing that
    is a separate and hard to solve problem. Original code also mmaped region
    as RWX, so this PR does not make things worse at least.

    Closes: https://github.com/open-power/skiboot/issues/258
    Signed-off-by: Georgy Yakovlev <email address hidden>
    Reviewed-by: Vasant Hegde <email address hidden>
    [oliver: whitespace fix, add a comment, reflow commit message]
    Signed-off-by: Oliver O'Halloran <email address hidden>

-Vasant

[Matthieu Clemenceau (mclemenceau)]

Hello,
I've uploaded a new version of skiboot for hirsute to this ppa ppa:mclemenceau/distro-work
Can you confirm this resolve the issue on this LP and I'll start release process for hirsute and other impacted series
Thanks
Matthieu

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-10 01:17 EDT-------
(In reply to comment #5)
> Hello,
> I've uploaded a new version of skiboot for hirsute to this ppa
> ppa:mclemenceau/distro-work
> Can you confirm this resolve the issue on this LP and I'll start release
> process for hirsute and other impacted series
> Thanks
> Matthieu

Looks good. Thanks!

-Vasant

[Matthieu Clemenceau (mclemenceau)]

Excellent, Can you also validate the solution for groovy and focal from this ppa?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-11 00:22 EDT-------
(In reply to comment #7)
> Excellent, Can you also validate the solution for groovy and focal from this
> ppa?

Looks good.

-Vasant

[Matthieu Clemenceau (mclemenceau)]

The attached patch should resolve this bug (LP: #1905393) as well as (LP: #1904585)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-14 06:20 EDT-------
(In reply to comment #10)
> Created attachment 146860 [details]
> skiboot_6.6.2-1_6.6.2-1ubuntu1.diff
>
> default comment

Looks good.

-Vasant

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Launchpad Janitor (janitor)]

This bug was fixed in the package skiboot - 6.6.2-1ubuntu1

---------------
skiboot (6.6.2-1ubuntu1) hirsute; urgency=medium

  * opal-prd: mmap(range:ibm,hbrt-code-image,...) fails with EPERM
     (LP: #1905393) opal-prd fails to start
     d/patches/0005-fix-opal-prd-fail-with-EPERM.patch
  * opal-prd: Have a worker process handle page offlining
     (LP: #1904585) Have a worker process handle page offlining
     d/patches/0006-fix-opal-prd-have-worker-process-handle-page-offlining.patch

 -- Matthieu Clemenceau <email address hidden> Mon, 07 Dec 2020 17:50:16 -0600

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted skiboot into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/skiboot/6.5.2-1ubuntu0.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted skiboot into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/skiboot/6.5.2-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-01-12 01:25 EDT-------
(In reply to comment #20)
> Hello bugproxy, or anyone else affected,
>
> Accepted skiboot into groovy-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/skiboot/6.5.2-1ubuntu0.20.10.1 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to
> enable and use -proposed. Your feedback will aid us getting this update out
> to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, what testing has been
> performed on the package and change the tag from verification-needed-groovy
> to verification-done-groovy. If it does not fix the bug for you, please add
> a comment stating that, and change the tag to verification-failed-groovy. In
> either case, without details of your testing we will not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s) fixed
> by this package have been verified and the package has been in -proposed for
> a minimum of 7 days.

Thanks! Changes looks good.

With this change we are able to start opal-prd daemon and co-worker fix also working fine.

-Vasant

[Launchpad Janitor (janitor)]

This bug was fixed in the package skiboot - 6.5.2-1ubuntu0.20.10.1

---------------
skiboot (6.5.2-1ubuntu0.20.10.1) groovy; urgency=medium

  * opal-prd: mmap(range:ibm,hbrt-code-image,...) fails with EPERM
     (LP: #1905393) opal-prd fails to start
     d/patches/0005-fix-opal-prd-fail-with-EPERM.patch
  * opal-prd: Have a worker process handle page offlining
     (LP: #1904585) Have a worker process handle page offlining
     d/patches/0006-fix-opal-prd-have-worker-process-handle-page-offlining.patch

 -- Matthieu Clemenceau <email address hidden> Mon, 07 Dec 2020 16:03:03 -0600

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for skiboot has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package skiboot - 6.5.2-1ubuntu0.20.04.1

---------------
skiboot (6.5.2-1ubuntu0.20.04.1) focal; urgency=medium

  * opal-prd: mmap(range:ibm,hbrt-code-image,...) fails with EPERM
     (LP: #1905393) opal-prd fails to start
     d/patches/0005-fix-opal-prd-fail-with-EPERM.patch
  * opal-prd: Have a worker process handle page offlining
     (LP: #1904585) Have a worker process handle page offlining
     d/patches/0006-fix-opal-prd-have-worker-process-handle-page-offlining.patch

 -- Matthieu Clemenceau <email address hidden> Mon, 07 Dec 2020 16:03:03 -0600