opal-prd not installed by default on ppc64el systems

[Jeremy Kerr]

[Impact]

 * opal-prd should be installed on OpenPOWER systems

[Test Case]

 * install power system

 * check that /sys/firmware/devicetree/base/ibm,opal/diagnostics exists

 * if above is true, check that opal-prd package is installed

[Regression Potential]

 * Additional package installation is performed, thus the install process may take longer, and additional disk space will be used. However the new code path is non-fatal, and all errors are ignored, thus all installation should still proceed past this point. For systems without relevant diagnostics exposed, this code path is a no-op.

[Other Info]

 * Original bug report

Just tried an install with current 16.04 network media, using standard server package selections, and it looks like opal-prd isn't installed by default:

 [jk@fstn ~]$ dpkg -l opal-prd
 dpkg-query: no packages found matching opal-prd

This is required for RAS-type functions on OpenPOWER machines; and has a similar role to something like acpid, on x86.

I'm not sure whether filing this against the skiboot package is best, or whether this should be moved to something installer-related. Happy to shift if necessary.

[Frédéric Bonnard (frediz)]

I think this is related to ubuntu-meta that provides the different meta packages for the different ubuntu flavors

[Frédéric Bonnard (frediz)]

Could you review the proposed patch based on the following :
some people recently ran into several issues on Minsky where they found that Opal PRD
was not running out on customer systems. "This is critical to our ability to
collect logs, determine what is causing failures, and for hostboot to take
appropriate action in regards to a failure."

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "1555904.debdiff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Breno Leitão (breno-leitao)]

I would like to mention that this is a very important package for OpenPower machine. We need to have it installed on every OpenPower machine that has Ubuntu.

[Steve Langasek (vorlon)]

The proposed patch (which we wouldn't take directly, since ubuntu-meta is generated from the seeds, but the intent is clear) would cause opal-prd to be installed on all ppc64el server installs, including cloud images (booted in KVM), PowerNV installs, and PowerVM installs. This doesn't sound quite like what we want here, is it?

What would be the right install-time platform check to decide whether or not we should have the opal-prd package included?

[Frédéric Bonnard (frediz)]

Hi Steve,
indeed, not all the machines would be concerned, even if the daemon won't be loaded as it already does
this check itself, which is why I was not sure to do the check at install time or later by the daemon itself.
It's better though to target only the specific configuration.
The criteria is the existence of /sys/firmware/devicetree/base/ibm,opal/diagnostics directory.

That is an openpower baremetal machine (ibmpower do not have that diagnostic feature accessible from the host as the fsp does the diagnotic work)
So yes the fix is to be done somewhere else then but I'm not sure where. Some udeb, tasksel, the installer?
Let me know and I can do a patch that you could review.
Thanks,

F.

[Steve Langasek (vorlon)]

> The criteria is the existence of
> /sys/firmware/devicetree/base/ibm,opal/diagnostics directory.

> That is an openpower baremetal machine (ibmpower do not have that
> diagnostic feature accessible from the host as the fsp does the diagnotic
> work) So yes the fix is to be done somewhere else then but I'm not sure
> where. Some udeb, tasksel, the installer? Let me know and I can do a
> patch that you could review. Thanks,

I believe this belongs in the hw-detect udeb for the installer.

[Adam Conrad (adconrad)]

I've filed https://bugs.launchpad.net/ubuntu/+source/skiboot/+bug/1691557 which will be a prerequisite for getting this seeded and installed via hw-detect.

[Steve Langasek (vorlon)]

The MIR in https://bugs.launchpad.net/ubuntu/+source/skiboot/+bug/1691557 has been approved, so this is now unblocked for the changes to be made in hw-detect and the seed. I'll update the seed immediately.

[Launchpad Janitor (janitor)]

This bug was fixed in the package hw-detect - 1.117ubuntu5

---------------
hw-detect (1.117ubuntu5) bionic; urgency=medium

  * hw-detect.sh: install opal-prd on OpenPOWER machines. LP: #1555904

 -- Dimitri John Ledkov <email address hidden> Wed, 29 Nov 2017 12:55:19 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package debian-installer - 20101020ubuntu525

---------------
debian-installer (20101020ubuntu525) bionic; urgency=medium

  * Move master kernels to 4.13.0-17.
  * Rebuild with new hw-detect LP: #1555904.

 -- Dimitri John Ledkov <email address hidden> Thu, 30 Nov 2017 11:01:35 +0000

[Dimitri John Ledkov (xnox)]

Please test d-i netboot installer 20101020ubuntu525 or later; or tomorrow's server images with a timestamp of 1st of December or later. It should resolve your issue. Please comment if your testing was successful. Then we can look into cherry-picking this fix for xenial upcoming point release too.

[Jeremy Kerr (jk-ozlabs)]

Just attempted with bionic 20101020ubuntu525 netboot installer. However, this build fails to install on my machine:

  │ │ Debootstrap warning │ │
  │ │ Warning: Failure trying to run: chroot /target dpkg-deb -f │ │
  │ │ /var/cache/apt/archives/dpkg_1.19.0.4ubuntu1_ppc64el.deb Version │ │

syslog tells me:

  Dec 11 08:58:46 debootstrap: chroot: can't execute 'dpkg-deb': No such file or directory

I can't run anything under /target:

  ~ # chroot /target /bin/bash
  chroot: can't execute '/bin/bash': No such file or directory
  ~ # ls -l /target/bin/bash
  -rwxr-xr-x 1 root root 1544376 May 17 2017 /target/bin/bash

Looks like that build has an invalid dynamic linker path, or the installer hasn't provided the requested linker yet:

  bash: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=8b1909e1534e08ac77ea8f8f1c8a120c6d02d631, stripped

  ~ # ls -al /target/lib64/
  drwxr-xr-x 2 root root 4096 Oct 11 20:21 .
  drwxr-xr-x 20 root root 4096 May 23 2017 ..

(I do see a suitable linker under /target/lib/powerpc64le-linux-gnu/ though)

[Jeremy Kerr (jk-ozlabs)]

I see this earlier in the installer syslog:

Dec 11 08:58:45 debootstrap: tar: skipping unsafe symlink to '/lib/powerpc64le-linux-gnu/ld-2.26.so' in archive, set EXTRACT_UNSAFE_SYMLINKS=1 to extract

(along with another bunch of the same error for different symlinks).

I can provide the full syslog if required.

[Jeremy Kerr (jk-ozlabs)]

I've filed that bug as https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1737661

[Robie Basak (racb)]

Hello Jeremy, or anyone else affected,

Accepted hw-detect into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/hw-detect/1.117ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

@IBM

Do you have any systems that have /sys/firmware/devicetree/base/ibm,opal/diagnostics and can test this on?

You can use the 16.04 xenial proposed d-i, and boot that with 'apt-setup/proposed=true' extra kernel cmdline this will use hw-detect from xenial-proposed, and should have this issue resolved, and automatically detect /diagnostics and thus install opal-prd in the target systems.

[Dimitri John Ledkov (xnox)]

URL to xenial-proposed d-i is http://ports.ubuntu.com/dists/xenial-proposed/main/installer-ppc64el/

[Dimitri John Ledkov (xnox)]

Unable to verify if the issue at hand is fixed or not, however code wise it does not have options to regress installations. The installation went fine on systems, that do not expose said diagnostics directory with hw-detect 1.117ubuntu2.3. Further testing will confirm if this issue is fixed, and can be closed.

[Frédéric Bonnard (frediz)]

Hi Dimitri,
I did the test with the netboot vmlinux/initrd.gz from xenial-proposed and 'apt-setup/proposed=true'
and it worked :

---
ubuntu@garri:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
ubuntu@garri:~$ apt-cache policy opal-prd
opal-prd:
  Installed: 5.4.3-1ubuntu0.16.04.1
  Candidate: 5.4.3-1ubuntu0.16.04.1
  Version table:
 *** 5.4.3-1ubuntu0.16.04.1 500
        500 http://de.ports.ubuntu.com/ubuntu-ports xenial-updates/main ppc64el Packages
        100 /var/lib/dpkg/status
     5.1.13-0ubuntu2 500
        500 http://de.ports.ubuntu.com/ubuntu-ports xenial/universe ppc64el Packages
ubuntu@garri:~$ ps axf|grep opal-prd
  1837 ? Ss 0:00 /usr/sbin/opal-prd --pnor /dev/mtd0
  2067 hvc0 S+ 0:00 \_ grep --color=auto opal-prd
ubuntu@garri:~$ ls -ld /sys/firmware/devicetree/base/ibm,opal/diagnostics
drwxr-xr-x 2 root root 0 Feb 23 14:55 /sys/firmware/devicetree/base/ibm,opal/diagnostics
---

Thanks,

F.

[Launchpad Janitor (janitor)]

This bug was fixed in the package hw-detect - 1.117ubuntu2.3

---------------
hw-detect (1.117ubuntu2.3) xenial; urgency=medium

  * hw-detect.sh: install opal-prd on OpenPOWER machines. LP: #1555904
  * Drop hw-detect.pre-pkgsel.d/20install-hwpackages, as it installs
    universe package, which is probably in itself not useful on
    Ubuntu. LP: #1577833

 -- Dimitri John Ledkov <email address hidden> Wed, 21 Feb 2018 15:36:54 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for hw-detect has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Jeremy, or anyone else affected,

Accepted debian-installer into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu451.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package debian-installer - 20101020ubuntu451.23

---------------
debian-installer (20101020ubuntu451.23) xenial; urgency=medium

  * No change rebuild to pick up the hw-detect changes (LP: #1555904).

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 28 Feb 2018 14:54:36 +0100