Ubuntu 20.04: opal-prd fails to start on 20.04
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
The Ubuntu-power-systems project |
Fix Released
|
Critical
|
Ubuntu on IBM Power Systems Bug Triage | ||
skiboot (Ubuntu) |
Fix Released
|
High
|
Matthieu Clemenceau | ||
Focal |
Fix Released
|
High
|
Matthieu Clemenceau | ||
Groovy |
Fix Released
|
High
|
Matthieu Clemenceau | ||
Hirsute |
Fix Released
|
High
|
Matthieu Clemenceau |
Bug Description
[Impact]
opal-prd (the daemon on a power system that listens for hw diagnostic system events at the /dev/opal-prd device) fails to start.
The reason is that opal-prd is not able to properly handle devtmpfs, mounted with noexec in /dev, which is the case on recent versions of systemd (like used in focal or newer)..
Currently such a mount prevents mapping HBRT image code region as 'rwx' from /dev.
[Fix]
This patch/commit attempts to work around the situation by copying HBRT image to a non mmapped memory region and sets mprotect rwx on it, allowing opal-prd to successfully execute the code region (as suggested here: https:/
47005e8d4c9aeda
[Test Case]
Since the opal-prd daemon must be running in the background as a separate process, the test is to:
- install the updated package that includes the patched opal-prd daemon (e.g. from the PPA mentioned below)
- double check the installed package version (dpkg -l) and maybe the opal-pd version that's in place (opal-prd --version)
- start opal-prd as daemon: 'service opal-prd start' (if not started automatically)
- verify the opal-prd status and check if it's running or not, by for example 'service opal-prd status'
[What could go wrong]
Things can go wrong in case the HBRT image copy is done wrong; in case it's accidentally copied to a wrong memory area (e.g. to an already mapped range, or erroneously calculated address/size), a seg. fault will happen and the system would core dump.
The mprotect code is pretty straight forward, but the fact that mprotect rwx is set on it, allows opal-prd to successfully execute the code region. It's not generally a perfect approach to map memory as RWX, but HBRT requires the ability to write into the image at runtime - and it got upstream accepted that way with skiboot v6.7.
The fix was released back in October and was pre-tested by the IBM Power team.
On top a patched Ubuntu package was build and shared in a PPA (see comment #1) and again successfully validated on focal and groovy.
__________
[Original Description]
== Comment: #0 - VASANT HEGDE <email address hidden> - 2020-11-23 23:23:22 ==
---Problem Description---
opal-prd fails to start on 20.04
Contact Information = Vasant hegde <email address hidden>
---uname output---
Ubuntu 20.04
Machine Type = All Power System
---Steps to Reproduce---
opal-prd fails to start on 20.04
Userspace tool common name: opal-prd
The userspace tool has the following bit modes: 64bit
Userspace rpm: opal-prd
This is fixed in upstream by below commit. Please backport this patch to 20.04 LTS release. Also applicable for 20.10.
commit 47005e8d4c9aeda
Author: Georgy Yakovlev <email address hidden>
Date: Mon Oct 12 14:29:17 2020 -0700
opal-prd: handle devtmpfs mounted with noexec
On systems using recent versions of systemd /dev (devtmpfs) is mounted with
noexec option. Such mount prevents mapping HBRT image code region as RWX
from /dev. This commit, as suggested in github PR linked below, attempts to
work around the situation by copying HBRT image to anon mmaped memory
region and sets mprotect rwx on it, allowing opal-prd to sucessfully
execute the code region.
Having memory region set as RWX is not ideal for security, but fixing that
is a separate and hard to solve problem. Original code also mmaped region
as RWX, so this PR does not make things worse at least.
Closes: https:/
Signed-off-by: Georgy Yakovlev <email address hidden>
Reviewed-by: Vasant Hegde <email address hidden>
[oliver: whitespace fix, add a comment, reflow commit message]
Signed-off-by: Oliver O'Halloran <email address hidden>
-Vasant
tags: | added: architecture-ppc64le bugnameltc-189959 severity-critical targetmilestone-inin20041 |
Changed in ubuntu: | |
assignee: | nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
affects: | ubuntu → opal (Ubuntu) |
Changed in ubuntu-power-systems: | |
assignee: | nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
importance: | Undecided → Critical |
Changed in opal (Ubuntu Hirsute): | |
assignee: | Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Foundations Team (canonical-foundations) |
tags: | added: fr-965 |
affects: | opal (Ubuntu Focal) → skiboot (Ubuntu Focal) |
Changed in skiboot (Ubuntu Groovy): | |
assignee: | nobody → Matthieu Clemenceau (mclemenceau) |
Changed in skiboot (Ubuntu Focal): | |
assignee: | nobody → Matthieu Clemenceau (mclemenceau) |
Changed in skiboot (Ubuntu Hirsute): | |
assignee: | Canonical Foundations Team (canonical-foundations) → Matthieu Clemenceau (mclemenceau) |
Changed in ubuntu-power-systems: | |
status: | New → Triaged |
Changed in skiboot (Ubuntu Hirsute): | |
status: | New → In Progress |
Changed in skiboot (Ubuntu Focal): | |
importance: | Undecided → High |
Changed in skiboot (Ubuntu Groovy): | |
importance: | Undecided → High |
Changed in skiboot (Ubuntu Hirsute): | |
importance: | Undecided → High |
Changed in ubuntu-power-systems: | |
status: | Triaged → In Progress |
Changed in skiboot (Ubuntu Focal): | |
milestone: | none → ubuntu-20.04.2 |
Changed in skiboot (Ubuntu Groovy): | |
status: | New → In Progress |
Changed in skiboot (Ubuntu Focal): | |
status: | New → In Progress |
Changed in skiboot (Ubuntu Hirsute): | |
status: | In Progress → Fix Committed |
description: | updated |
Changed in skiboot (Ubuntu Focal): | |
milestone: | ubuntu-20.04.2 → focal-updates |
description: | updated |
Changed in ubuntu-power-systems: | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done verification-done-focal removed: verification-needed verification-needed-focal |
Changed in ubuntu-power-systems: | |
status: | Fix Committed → Fix Released |
Hello, /distro- work
I've uploaded a new version of skiboot for hirsute to this ppa ppa:mclemenceau
Can you confirm this resolve the issue on this LP and I'll start release process for hirsute and other impacted series
Thanks
Matthieu