CPI information about kvm usage is lost during apt upgrade

[bugproxy]

SRU Justification:

[Impact]

 * Needlessly slow systemctl daemon-reload on baremetal s390x machines
 * Wipes information about auxilary bits, by force resetting kernel version

[Test Case]

$ cat /sys/firmware/cpi/system_level
0x0000000000041200

Note the 41200 digits, this is hex-encoded kernel version number. Can be different depending on the kernel in use

$ sudo /lib/s390-tools/cpictl -b kvm
$ cat /sys/firmware/cpi/system_level
0x8000000000041200

# Note the leading bit is now 0x8, instead of 0x0

$ sudo systemctl daemon-reload
$ cat /sys/firmware/cpi/system_level
0x8000000000041200 -> is correct, 0x8 is preserved
0x0000000000041200 -> is wrong, 0x8 got reset to 0x0

[Solution]
In Ubuntu, cpi vars are set using systemd generator on early boot, thus HMC displays correct information about an LPAR very early on.
However, setting them takes about 1s, and at the moment was done upon every systemctl daemon-reload.
In addition, when kvm is used, cplictl -b kvm is called to update the leading kvm usage bit, this was getting lost upon every systemctl daemon-reload.

Instead, modify the generator to guard against re-setting cpi vars, if they were ever set once.

[Regression Potential]

 * At worst case scenario, cpi-vars would not be set, leading to cosmetic issue of not able to see auxilary information (e.g. UBUNTU, running kernel version, Ubuntu release version) in the HMC console. As the variables form /sys/firmware/cpi/* are displayed in the HMC.

[Other Info]

 * Original bug report

---Problem Description---
CPI information about kvm usage is lost during apt upgrade

---uname output---
4.15.0-42-generic

Machine Type = s390x lpar

---Steps to Reproduce---

# cat /sys/firmware/cpi/*
cat: /sys/firmware/cpi/set: Permission denied

0x0000000000040f00

LINUX
root@m83lp52:~# virsh start guest # starting a guest does set the highest bit in
/sys/firmware/cpi/system_leveln

root@m83lp52:~# cat /sys/firmware/cpi/*
cat: /sys/firmware/cpi/set: Permission denied

0x8000000000040f00 <---- see the 8

LINUX

root@m83lp52:~# apt upgrade
[....]
has to upgrade at least one package, that will trigger some other code.
[....]

Now. some other code seems to have set values in /sys/firmware/cpi as well dropping the kvm information.

root@m83lp52:~# cat /sys/firmware/cpi/*
cat: /sys/firmware/cpi/set: Permission denied
18 04 1
0x0000000000040f00
UBUNTU
LINUX

The canonical place to handle /sys/firmware/cpi should be /lib/s390-tools/cpictl and /lib/systemd/system/cpi.service. So can we put the additional features (setting 18 04 1 and UBUNTU things also into
/etc/sysconfig/cpi
or
/lib/s390-tools/cpictl
)
and can we disable the other code that also sets /sys/firmware/cpi (I have not found out which code sets these values). This seems to be /lib/systemd/system-generators/s390-cpi-vars

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-11-29 09:18 EDT-------
removing /lib/systemd/system-generators/s390-cpi-vars Seems to help. (this seems to be an ubuntu specific extension).
We just need to move this functionality (adding ubuntu and the version) into the common cpi handling.

[Dimitri John Ledkov (xnox)]

/lib/systemd/system-generators/s390-cpi-vars correct is how I implemented cpi-vars before cpictl was avaiable.

/etc/sysconfig is RPM-ism which is generally should not be used on Ubuntu (or to be fair anywhere these days).

I do not like specifying or hardcoding things in a "one-more-config file".

Imho instead cpictl by default should use the distro-independant files to echo sensible things for any Linux based on e.g. /usr/lib/os-release /etc/os-release /etc/lsb-release and the like, as shipped in ubuntu/debian by base-files packages and similar packages on all other distributions. Such that, without hardcoding anything, obviously correct and self-identifying things are set.

It has been on my backlog to migrate to cpictl, but I have not done so.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-11-30 07:54 EDT-------
The problem is now, that /lib/systemd/system-generators/s390-cpi-vars overwrites some values that are set by cpictl. (for example the KVM token).

Would a quick fix be, that s390-cpi-vars does everything as of today, but leaves the version code unchanged?
So something like
--- /lib/systemd/system-generators/s390-cpi-vars.old 2018-11-30 13:53:45.525120681 +0100
+++ /lib/systemd/system-generators/s390-cpi-vars 2018-11-30 13:54:26.875220713 +0100
@@ -3,6 +3,5 @@
. /etc/os-release
echo "$NAME" >/sys/firmware/cpi/system_name
echo "LINUX" >/sys/firmware/cpi/system_type
-printf "0x%02x%02x%02x" $(uname -r | sed 's/-.*//;s/\./ /g') >/sys/firmware/cpi/system_level
echo "$VERSION" | sed 's/ .*//;s/\./ /g' >/sys/firmware/cpi/sysplex_name
echo 1 >/sys/firmware/cpi/set

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2018-12-03 08:05 EDT-------

This would be a quick fix that let cpictl fully handle the kernel level, while s390-cpi-vars would still be active as today for the remaining things.
Works for me.

Of course this is only a partial step that restores some of the cpictl functionality. Long term we must integrate s390-cpi-vars somehow in cpictl.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-08 08:36 EDT-------
ping. Any progress on the minimal fix for now?

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 2.8.0-0ubuntu4

---------------
s390-tools (2.8.0-0ubuntu4) disco; urgency=medium

  * Guard against needlesly resetting cpi vars. LP: #1805841

 -- Dimitri John Ledkov <email address hidden> Wed, 13 Mar 2019 23:44:33 +0000

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-15 06:04 EDT-------
This is for the s390-tools in disco. Can we backport that fix also into the s390-tools 2.3 of bionic?

[Frank Heimes (fheimes)]

Yes, that's in progress - indicated by the SRU Justification that was added to the bug description.
This is part of the SRU process and always needed in case of SRUing or backporting things back to already released Ubuntu versions.
Such things need to be processed starting with the current dev. release (which is disco here) going back to the 'oldest' release the patch should land (in this case bionic).

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2018-12-03 08:05 EDT-------

This would be a quick fix that let cpictl fully handle the kernel level, while s390-cpi-vars would still be active as today for the remaining things.
Works for me.

Of course this is only a partial step that restores some of the cpictl functionality. Long term we must integrate s390-cpi-vars somehow in cpictl.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-04-09 06:57 EDT-------
Any update on the SRU process for bionic?

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted s390-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.3.0-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-05-17 04:53 EDT-------
I can confirm that
http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x s390-tools s390x 2.3.0-0ubuntu3.3
does fix the issue.
(verified with systemctl daemon-reload. This triggered the problem reliably)

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted s390-tools into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.6.0-0ubuntu7.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for s390-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 2.3.0-0ubuntu3.3

---------------
s390-tools (2.3.0-0ubuntu3.3) bionic; urgency=medium

  * Guard against needlesly resetting cpi vars. LP: #1805841
  * Cherrypick upstream patch to add OSA-Express7S defines in
    qethqoat. LP: #1815022

 -- Dimitri John Ledkov <email address hidden> Thu, 02 May 2019 14:50:47 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-05-23 07:56 EDT-------
Successfully verified on cosmic with
s390-tools s390x 2.6.0-0ubuntu7.3

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 2.6.0-0ubuntu7.3

---------------
s390-tools (2.6.0-0ubuntu7.3) cosmic; urgency=medium

  * Guard against needlesly resetting cpi vars. LP: #1805841
  * Use system path to find cryptsetup binary. LP: #1803958
  * Cherrypick upstream patch to add OSA-Express7S defines in
    qethqoat. LP: #1815022

 -- Dimitri John Ledkov <email address hidden> Thu, 02 May 2019 14:21:10 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-06-11 02:33 EDT-------
IBM bugzilla status ->closed, Fix Released for all requested distros